Oct 29 22:39:46.267465 ip-10-0-139-130 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 29 22:39:46.267477 ip-10-0-139-130 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 29 22:39:46.267486 ip-10-0-139-130 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 29 22:39:46.267796 ip-10-0-139-130 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 29 22:39:56.516800 ip-10-0-139-130 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 29 22:39:56.516819 ip-10-0-139-130 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ac7770c561d649e4b45c9831064b2b70 --
Oct 29 22:41:42.884387 ip-10-0-139-130 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:41:43.275116 ip-10-0-139-130 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:41:43.275116 ip-10-0-139-130 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:41:43.275116 ip-10-0-139-130 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:41:43.275116 ip-10-0-139-130 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:41:43.278417 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278341    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:43.278417 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278416    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:43.278417 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278419    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:43.278417 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278423    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278426    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278429    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278432    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278435    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278437    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278440    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278442    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278446    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278448    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278451    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278454    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278456    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278464    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278467    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278470    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278473    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278475    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278478    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278481    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:43.278555 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278483    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278486    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278488    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278491    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278494    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278496    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278499    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278501    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278504    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278506    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278509    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278511    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278514    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278517    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278520    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278523    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278525    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278528    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278530    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278533    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:43.279012 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278535    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278538    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278541    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278544    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278546    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278549    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278552    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278554    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278557    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278560    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278562    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278566    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278570    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278575    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278578    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278581    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278584    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278586    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278589    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:43.279493 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278592    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278594    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278597    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278600    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278603    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278606    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278608    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278611    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278613    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278616    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278618    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278621    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278625    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278629    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278632    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278636    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278639    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278642    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278644    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:43.279940 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278647    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278649    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278652    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278654    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278657    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278659    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278662    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278664    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278667    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278670    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278672    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278675    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278677    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278680    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278683    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278685    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278687    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278691    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278695    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:43.280409 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278697    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278699    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278702    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278706    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278708    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.278711    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279489    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279496    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279499    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279502    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279505    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279509    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279512    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279515    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279517    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279520    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279523    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279525    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279528    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279530    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:43.280891 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279533    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279535    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279538    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279540    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279543    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279546    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279549    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279552    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279555    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279558    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279560    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279563    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279565    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279570    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279573    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279576    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279578    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279581    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279584    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279586    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:43.281378 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279589    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279591    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279594    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279596    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279599    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279602    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279604    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279607    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279609    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279612    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279614    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279617    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279620    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279623    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279626    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279630    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279635    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279638    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279641    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:43.281852 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279644    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279647    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279650    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279652    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279655    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279657    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279660    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279663    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279665    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279668    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279670    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279673    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279675    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279678    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279680    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279683    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279685    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279688    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279691    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279693    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:43.282312 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279696    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279698    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279701    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279703    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279706    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279709    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279711    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279714    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279716    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279719    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279721    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279724    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279726    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279729    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279732    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279735    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279737    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279740    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279743    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:43.282788 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279745    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279748    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279750    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279753    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279756    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279758    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279760    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279763    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279766    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279769    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279771    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279774    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279777    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.279779    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280335    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280342    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280345    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280348    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280350    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280353    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:43.283254 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280356    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280359    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280361    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280364    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280367    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280369    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280373    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280376    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280380    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280382    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280385    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280388    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280390    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280393    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280395    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280398    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280401    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280404    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280406    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280409    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:43.283719 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280411    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280414    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280417    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280420    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280423    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280426    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280429    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280431    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280434    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280436    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280439    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280442    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280444    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280447    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280449    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280452    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280455    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280457    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280460    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280463    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:43.284192 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280466    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280468    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280471    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280473    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280476    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280479    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280481    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280483    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280486    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280488    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280492    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280495    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280498    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280501    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280504    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280506    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280509    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280512    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280515    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:43.284675 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280517    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280520    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280523    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280525    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280528    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280530    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280533    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280535    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280538    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280541    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280544    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280547    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280549    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280553    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280555    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280558    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280560    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280563    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280567    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280571    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:43.285127 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280574    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280576    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280579    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280582    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280585    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280587    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280590    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280592    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280595    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280598    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280600    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280603    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280606    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280609    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280612    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280614    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280617    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280619    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280622    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280625    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:43.285631 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.280630    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281565    2582 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281576    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281582    2582 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281587    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281591    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281595    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281599    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281604    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281607    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281610    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281614    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281618    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281621    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281624    2582 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281627    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281630    2582 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281633    2582 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281636    2582 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281641    2582 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281644    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281647    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281650    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281653    2582 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:41:43.286099 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281658    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281661    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281665    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281668    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281671    2582 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281674    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281677    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281680    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281685    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281689    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281692    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281695    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281698    2582 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281701    2582 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281704    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281708    2582 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281711    2582 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281714    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281718    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281721    2582 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281726    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281729    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281732    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281735    2582 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281738    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:41:43.286678 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281741    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281744    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281748    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281751    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281753    2582 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281756    2582 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281760    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281763    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281767    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281770    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281773    2582 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281776    2582 flags.go:64] FLAG: --help="false"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281779    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281783    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281786    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281790    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281794    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281797    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281800    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281803    2582 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281806    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281809    2582 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281812    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281816    2582 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:41:43.287329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281819    2582 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281822    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281824    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281827    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281830    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281833    2582 flags.go:64] FLAG: --lock-file=""
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281836    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281838    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281842    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281847    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281850    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281853    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281856    2582 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281859    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281863    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281866    2582 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281869    2582 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281873    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281877    2582 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281880    2582 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281883    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281886    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281889    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281892    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281895    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:41:43.287930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281898    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281901    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281908    2582 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281911    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281914    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281917    2582 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281920    2582 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281923    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281926    2582 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281929    2582 flags.go:64] FLAG: --port="10250"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281932    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281935    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-084c925e6068d5185"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281938    2582 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281941    2582 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281944    2582 flags.go:64] FLAG: --register-node="true"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281947    2582 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281950    2582 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281953    2582 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281956    2582 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281959    2582 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281962    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281965    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281968    2582 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281971    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281974    2582 flags.go:64] FLAG: --runonce="false"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281977    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:41:43.288536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281981    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281984    2582 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281986    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281989    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281993    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281996    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.281999    2582 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282002    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282005    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282008    2582 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282010    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282015    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282018    2582 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282021    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282026    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282029    2582 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282032    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282038    2582 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282040    2582 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282043    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282046    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282049    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282052    2582 flags.go:64] FLAG: --v="2"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282057    2582 flags.go:64] FLAG: --version="false"
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282061    2582 flags.go:64] FLAG: --vmodule=""
Oct 29 22:41:43.289185 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282065    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282068    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282214    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282219    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282222    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282225    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282227    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282231    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282233    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282236    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282239    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282242    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282244    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282247    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282249    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282252    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282255    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282257    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282261    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282264    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:43.289856 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282267    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282269    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282272    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282274    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282277    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282280    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282282    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282285    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282287    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282290    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282292    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282295    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282297    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282300    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282302    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282305    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282307    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282310    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282313    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282315    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:43.290411 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282318    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282321    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282324    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282326    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282329    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282331    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282334    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282337    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282339    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282341    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282345    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282348    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282350    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282353    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282356    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282358    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282361    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282363    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282366    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282369    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:43.290876 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282371    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282374    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282376    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282379    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282382    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282384    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282387    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282389    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282393    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282396    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282399    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282401    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282404    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282407    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282411    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282414    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282417    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282420    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282423    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:43.291365 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282425    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282428    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282431    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282434    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282437    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282441    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282445    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282448    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282451    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282453    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282456    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282459    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282462    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282464    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282467    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282469    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282472    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282475    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282477    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:43.291817 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282480    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282483    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282486    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282489    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282492    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282494    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282497    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282499    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282502    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282505    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.282531    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282569    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282574    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282577    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282579    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282582    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:43.292268 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282584    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282588    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282591    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282594    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282596    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282600    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282602    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282605    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282607    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282610    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282613    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282615    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282618    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282621    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282624    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282626    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282629    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282631    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282635    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282638    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:43.292722 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282640    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282643    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282645    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282648    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282651    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282654    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282656    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282659    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282661    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282664    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282666    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282669    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282672    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282676    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282678    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282681    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282683    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282686    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282689    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282692    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:43.293214 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282694    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282697    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282699    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282702    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282705    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282707    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282710    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282712    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282715    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282718    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282721    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282724    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282726    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282729    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282732    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282734    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282737    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282739    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282742    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:43.293693 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282746    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282749    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282752    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282755    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282758    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282761    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282764    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282767    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282770    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282772    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282776    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282779    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282782    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282785    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282788    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282792    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282795    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282798    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282801    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:43.294158 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282804    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282807    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282810    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282813    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282817    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282819    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282822    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282825    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282827    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282830    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282832    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282835    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282837    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282847    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282850    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282853    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282855    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282858    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282861    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282865    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:43.294635 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282867    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282870    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.282872    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.289266    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.289281    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.289296    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.289300    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289358    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289364    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289367    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289370    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289373    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289375    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289378    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289381    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289384    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289388    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289393    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289396    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:43.295110 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289399    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289403    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289405    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289408    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289410    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289413    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289417    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289419    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289422    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289425    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289427    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289430    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289432    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289435    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289437    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289440    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289443    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289446    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289449    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289452    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:43.295576 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289454    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289457    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289460    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289463    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289465    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289468    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289470    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289473    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289475    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289478    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289481    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289483    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289486    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289489    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289491    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289494    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289496    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289499    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289501    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289504    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:43.296051 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289507    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289510    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289513    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289515    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289518    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289521    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289523    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289526    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289528    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289532    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289534    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289537    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289539    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289542    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289544    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289547    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289550    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289552    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289555    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:43.296536 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289557    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289560    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289562    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289565    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289568    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289570    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289572    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289575    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289577    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289580    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289582    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289585    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289587    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289590    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289593    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289596    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289598    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289601    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289603    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289606    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:43.296979 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289609    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289611    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289614    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289617    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289619    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289622    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289624    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289628    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289632    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289635    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289637    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289640    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289643    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289645    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289648    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:43.297474 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.289675    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289815    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289820    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289823    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289826    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289828    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289831    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289833    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289836    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289838    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289841    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289845    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289847    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289850    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289853    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289855    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289858    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289861    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289863    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289866    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:43.297829 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289869    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289872    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289874    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289877    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289880    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289883    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289885    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289888    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289890    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289893    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289895    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289898    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289900    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289903    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289905    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289908    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289910    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289912    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289915    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:43.298438 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289917    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289920    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289922    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289925    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289928    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289933    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289937    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289941    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289945    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289948    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289951    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289954    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289956    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289959    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289962    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289964    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289967    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289969    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289972    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:43.299247 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289974    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289977    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289979    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289982    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289984    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289987    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289989    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289991    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289994    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289997    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.289999    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290002    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290004    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290007    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290009    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290012    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290014    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290017    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290020    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:43.299930 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290023    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290025    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290028    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290030    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290032    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290035    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290037    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290040    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290043    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290045    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290048    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290050    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290052    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290055    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290057    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290060    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290063    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290065    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290067    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:43.300446 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290070    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290072    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290075    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290077    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290079    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290096    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290100    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290103    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290105    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290108    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290110    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.290133    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290171    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290175    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290179    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290181    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:43.300943 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290184    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290187    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290189    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290192    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290195    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290198    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290201    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290203    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290206    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290209    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290211    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290214    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290216    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290219    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290221    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290223    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290226    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290228    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290231    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:43.301404 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290233    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290236    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290239    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290241    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290244    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290246    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290249    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290252    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290254    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290256    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290259    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290261    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290265    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290268    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290270    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290273    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290275    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290278    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290280    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290283    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:43.301926 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290285    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290288    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290290    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290292    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290295    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290297    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290300    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290302    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290305    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290307    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290310    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290313    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290315    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290319    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290322    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290326    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290329    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290332    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290335    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:43.302444 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290338    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290341    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290344    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290346    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290349    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290352    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290359    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290361    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290364    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290367    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290369    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290372    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290374    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290377    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290379    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290382    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290385    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290387    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290390    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290393    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:43.302935 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290395    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290398    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290400    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290403    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290405    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290407    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290410    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290413    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290415    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290418    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290421    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290423    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290426    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290428    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290431    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290433    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290436    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290438    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290441    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:43.303507 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290444    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290447    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290450    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290452    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:43.290456    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.290506    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.292385    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.293351    2582 server.go:1014] "Starting client certificate rotation"
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.293447    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.294177    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.295886    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:41:43.303950 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.297388    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:41:43.308173 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.308159    2582 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:41:43.311304 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.311287    2582 log.go:25] "Validated CRI v1 image API"
Oct 29 22:41:43.312154 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.312123    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:41:43.316384 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.316367    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9884d022-d94c-42a5-93d8-442042369ed2:/dev/nvme0n1p4 e3eadb42-9847-4e78-b331-72bc77e4e699:/dev/nvme0n1p3]
Oct 29 22:41:43.316428 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.316384    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 22:41:43.321606 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.321493    2582 manager.go:217] Machine: {Timestamp:2026-10-29 22:41:43.320112286 +0000 UTC m=+0.315576386 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101156 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2d3dc445d683e9cd3a394b39d188 SystemUUID:ec2b2d3d-c445-d683-e9cd-3a394b39d188 BootID:ac7770c5-61d6-49e4-b45c-9831064b2b70 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c3:47:a1:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c3:47:a1:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:86:bd:87:be:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:41:43.322113 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.322103    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:41:43.322193 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.322181    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:41:43.324662 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.324636    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:41:43.324791 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.324663    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:41:43.324839 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.324793    2582 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:41:43.324839 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.324801    2582 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:41:43.324892 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.324841    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:41:43.324892 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.324858    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:41:43.325618 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.325608    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:41:43.329159 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.329147    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:41:43.329266 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.329257    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:41:43.329969 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.329961    2582 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:41:43.330000 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.329974    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:41:43.330000 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.329983    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:41:43.330000 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.329992    2582 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:41:43.330078 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.330001    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:41:43.330312 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.330301    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:41:43.333110 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.333093    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:41:43.337623 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.337608    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:41:43.338324 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.338309    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:41:43.338401 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.338350    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:41:43.338550 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.338537    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:41:43.338607 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.338556    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:41:43.338607 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.338565    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:41:43.338607 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.338574    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:41:43.338607 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.338582    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:41:43.338607 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.338591    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:41:43.339208 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.339197    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:41:43.339270 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.339211    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:41:43.339270 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.339222    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:41:43.339270 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.339233    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:41:43.339270 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.339253    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:41:43.339270 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.339266    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:41:43.340911 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.340899    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:41:43.340967 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.340914    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:41:43.344465 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.344448    2582 server.go:1263] "Started kubelet"
Oct 29 22:41:43.344554 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.344501    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:41:43.348307 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.348129    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:41:43.348307 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.348225    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:41:43.348960 ip-10-0-139-130 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:41:43.349546 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.349527    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:41:43.352613 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.352597    2582 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:41:43.354763 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.354743    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 22:41:43.355899 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.355875    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:41:43.356366 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.356348    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:41:43.356571 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.356544    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:43.357129 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357023    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:41:43.357129 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357022    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:41:43.357129 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357055    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:41:43.357277 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357140    2582 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:41:43.357277 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357151    2582 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:41:43.357370 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357355    2582 factory.go:55] Registering systemd factory
Oct 29 22:41:43.357370 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357372    2582 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:41:43.357612 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357595    2582 factory.go:153] Registering CRI-O factory
Oct 29 22:41:43.357612 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357612    2582 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:41:43.357684 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357660    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:41:43.357684 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357677    2582 factory.go:103] Registering Raw factory
Oct 29 22:41:43.357740 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.357690    2582 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:41:43.358537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.358526    2582 manager.go:319] Starting recovery of all containers
Oct 29 22:41:43.367988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.367973    2582 manager.go:324] Recovery completed
Oct 29 22:41:43.373945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.373015    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:43.374862 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.374830    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:43.374947 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.374878    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:43.374947 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.374892    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:43.375457 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.375441    2582 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:41:43.375457 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.375455    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:41:43.375554 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.375445    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:43.375554 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.375474    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:41:43.376154 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.376137    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:43.376246 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.376191    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:43.376246 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.376221    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:43.377607 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.377595    2582 policy_none.go:50] "Start"
Oct 29 22:41:43.377672 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.377610    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:41:43.377672 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.377624    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:41:43.379667 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.379653    2582 policy_none.go:44] "Start"
Oct 29 22:41:43.381593 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.381572    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-130.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 22:41:43.381593 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.381573    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 22:41:43.382467 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.381580    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-130.ec2.internal.18e3214fd2af3b13  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-130.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-130.ec2.internal,},FirstTimestamp:2026-10-29 22:41:43.344421651 +0000 UTC m=+0.339885750,LastTimestamp:2026-10-29 22:41:43.344421651 +0000 UTC m=+0.339885750,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-130.ec2.internal,}"
Oct 29 22:41:43.391948 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.391931    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:41:43.395488 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.395400    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-130.ec2.internal.18e3214fd47faecd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-130.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-130.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-130.ec2.internal,},FirstTimestamp:2026-10-29 22:41:43.374859981 +0000 UTC m=+0.370324083,LastTimestamp:2026-10-29 22:41:43.374859981 +0000 UTC m=+0.370324083,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-130.ec2.internal,}"
Oct 29 22:41:43.395881 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.395859    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p7rqz"
Oct 29 22:41:43.395956 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.395908    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 22:41:43.395956 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.395914    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 29 22:41:43.395956 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.395908    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.403663    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p7rqz"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.415816    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.418065    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.419286    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.419305    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.419323    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.419352    2582 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.421050    2582 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.421178    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.421189    2582 server.go:87] "Starting device plugin registration server"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.421449    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.421462    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.421799    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.421867    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.421875    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.422304    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.422350    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:43.434162 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.433603    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:43.520861 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.520827    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"]
Oct 29 22:41:43.520971 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.520902    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:43.521599 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.521576    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:43.521896 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.521880    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:43.521952 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.521912    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:43.521952 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.521927    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:43.522297 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.522280    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:43.522386 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.522304    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:43.522386 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.522313    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:43.522386 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.522337    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.524151 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.524139    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:43.524295 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.524275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.524341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.524310    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:43.525074 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.525058    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:43.525178 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.525096    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:43.525178 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.525108    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:43.525178 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.525111    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:43.525178 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.525128    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:43.525178 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.525137    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:43.527217 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.527202    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.527299 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.527227    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:43.528153 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.528140    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:43.528209 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.528164    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:43.528209 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.528173    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:43.541329 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.541311    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.541425 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.541333    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-130.ec2.internal\": node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:43.559060 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.558882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.559060 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.558980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.559060 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.559010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67936acd58076934a9248bfd7519a6fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-130.ec2.internal\" (UID: \"67936acd58076934a9248bfd7519a6fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.562830 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.562813    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-130.ec2.internal\" not found" node="ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.565486 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.565470    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-130.ec2.internal\" not found" node="ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.587274 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.587253    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:43.659465 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.659438    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.659595 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.659387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.659595 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.659517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67936acd58076934a9248bfd7519a6fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-130.ec2.internal\" (UID: \"67936acd58076934a9248bfd7519a6fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.659595 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.659542    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.659595 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.659580    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.659729 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.659617    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67936acd58076934a9248bfd7519a6fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-130.ec2.internal\" (UID: \"67936acd58076934a9248bfd7519a6fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.687451 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.687431    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:43.787548 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.787484    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:43.866060 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.865893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.869692 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:43.868583    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:43.887763 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.887730    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:43.988366 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:43.988328    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:44.088842 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:44.088760    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:44.189602 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:44.189554    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:44.290521 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:44.290480    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:44.293708 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.293686    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 22:41:44.293866 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.293847    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:41:44.391309 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:44.391279    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:44.409955 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.409907    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:36:44 +0000 UTC" deadline="2028-07-04 13:15:36.212514102 +0000 UTC"
Oct 29 22:41:44.409955 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.409951    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14726h33m51.802566216s"
Oct 29 22:41:44.453392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.453366    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:44.454383 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.454366    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:44.454468 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.454398    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:44.454468 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.454410    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:44.477374 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:44.477344    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67936acd58076934a9248bfd7519a6fc.slice/crio-f7f0bfff4d3bf8d4b834e70e374998fc8468c3163770df33395314b16974182e WatchSource:0}: Error finding container f7f0bfff4d3bf8d4b834e70e374998fc8468c3163770df33395314b16974182e: Status 404 returned error can't find the container with id f7f0bfff4d3bf8d4b834e70e374998fc8468c3163770df33395314b16974182e
Oct 29 22:41:44.482275 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:44.482251    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63db5a61472622137957294ef9d38f93.slice/crio-8d0e760708084789f2b866adb72ceaf0c5ed24e8a749fc14429637c66bcaf20a WatchSource:0}: Error finding container 8d0e760708084789f2b866adb72ceaf0c5ed24e8a749fc14429637c66bcaf20a: Status 404 returned error can't find the container with id 8d0e760708084789f2b866adb72ceaf0c5ed24e8a749fc14429637c66bcaf20a
Oct 29 22:41:44.484382 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.484362    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:41:44.491874 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:44.491856    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:44.552785 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.552760    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:44.594214 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:44.594133    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:41:44.643654 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.643634    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:44.657721 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.657695    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:44.682214 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.682190    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:41:44.684719 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.684703    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:41:44.706037 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.706011    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:41:44.960907 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:44.960821    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:45.305465 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.305388    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:45.331840 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.331809    2582 apiserver.go:52] "Watching apiserver"
Oct 29 22:41:45.357490 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.357455    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 22:41:45.361786 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.361756    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:41:45.362202 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.362174    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-59ghk","openshift-multus/network-metrics-daemon-f47cd","openshift-network-diagnostics/network-check-target-j26t7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal","openshift-multus/multus-9rm5r","openshift-monitoring/node-exporter-tgl57","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx","openshift-ovn-kubernetes/ovnkube-node-dqkrw","kube-system/global-pull-secret-syncer-6vdmx","kube-system/konnectivity-agent-nfh24","openshift-dns/node-resolver-xrt59","openshift-multus/multus-additional-cni-plugins-qh8k5","openshift-cluster-node-tuning-operator/tuned-xj7f4","openshift-image-registry/node-ca-d55dr","kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"]
Oct 29 22:41:45.365271 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.365250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:45.367293 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.367273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.369663 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.369643    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:45.369749 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.369726    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:41:45.372067 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.372048    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.374942 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.374919    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.376898 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.376874    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.379277 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.379176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:45.379277 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.379248    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:41:45.381435 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.381279    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:41:45.381546 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.381365    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:41:45.381739 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.381719    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56msd\""
Oct 29 22:41:45.384930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.382189    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bd868\""
Oct 29 22:41:45.384930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.382527    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.384930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.382716    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.384930 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.382886    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.384930 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.382961    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:41:45.385472 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.385453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.388056 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.388035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.390605 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.390585    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.393215 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.393135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.395705 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.395674    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.413033 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.413006    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:41:45.413190 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.413076    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xtskk"
Oct 29 22:41:45.426259 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.426207    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal" event={"ID":"63db5a61472622137957294ef9d38f93","Type":"ContainerStarted","Data":"8d0e760708084789f2b866adb72ceaf0c5ed24e8a749fc14429637c66bcaf20a"}
Oct 29 22:41:45.427551 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.427531    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal" event={"ID":"67936acd58076934a9248bfd7519a6fc","Type":"ContainerStarted","Data":"f7f0bfff4d3bf8d4b834e70e374998fc8468c3163770df33395314b16974182e"}
Oct 29 22:41:45.459011 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.458984    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:41:45.469358 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-conf-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.469358 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.469552 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-log-socket\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.469552 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-kubelet-config\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.469552 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-lib-modules\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.469552 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.469729 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbwb\" (UniqueName: \"kubernetes.io/projected/09138ffd-6ca5-4c22-bca1-fac58f4849c1-kube-api-access-9nbwb\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.469729 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-netd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.469729 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-env-overrides\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.469890 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-sys\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.469890 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469789    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/549afed0-003b-43bf-8154-49e89fcba816-konnectivity-ca\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:45.469890 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-root\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.469890 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-multus\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.470026 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-cnibin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.470026 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-modprobe-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.470026 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469954    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-textfile\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.470026 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.469998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-system-cni-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.470026 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-bin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-multus-certs\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b117295a-3db5-4ec0-8858-43f561c7a075-serviceca\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-tmp-dir\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-kubernetes\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-var-lib-kubelet\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-system-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470202    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctk5g\" (UniqueName: \"kubernetes.io/projected/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-kube-api-access-ctk5g\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdc7\" (UniqueName: \"kubernetes.io/projected/ba102785-5e33-4eeb-9cc8-4f03627d328a-kube-api-access-dkdc7\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.470232 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470234    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-registration-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.470642 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-kubelet\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.470642 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-ovn\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.470642 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-wtmp\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.470642 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.470642 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470416    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9lf4\" (UniqueName: \"kubernetes.io/projected/729ef181-4607-453c-ae28-080401ddc444-kube-api-access-x9lf4\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:45.470642 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg46r\" (UniqueName: \"kubernetes.io/projected/b117295a-3db5-4ec0-8858-43f561c7a075-kube-api-access-fg46r\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.470642 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-sys-fs\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-node-log\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-config\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-dbus\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-run\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470846    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-sys\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-etc-kubernetes\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-host-slash\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b117295a-3db5-4ec0-8858-43f561c7a075-host\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.470988 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.470988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysconfig\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-host\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cnibin\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471089    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-cni-binary-copy\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-k8s-cni-cncf-io\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-slash\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-hosts-file\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-socket-dir-parent\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-daemon-config\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74d2\" (UniqueName: \"kubernetes.io/projected/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kube-api-access-b74d2\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-conf\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-tuned\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.471520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovn-node-metrics-cert\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-tls\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471556    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-device-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-etc-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-netns\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4spv\" (UniqueName: \"kubernetes.io/projected/1c02209b-22cf-4ed8-aa61-429052c91a19-kube-api-access-w4spv\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-systemd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-script-lib\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk92n\" (UniqueName: \"kubernetes.io/projected/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-kube-api-access-sk92n\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-tmp\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.472341 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-kubelet\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-hostroot\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-systemd-units\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.471985    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-netns\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-systemd\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-metrics-client-ca\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b29bj\" (UniqueName: \"kubernetes.io/projected/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-kube-api-access-b29bj\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-socket-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-var-lib-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-bin\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/549afed0-003b-43bf-8154-49e89fcba816-agent-certs\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-iptables-alerter-script\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5tmc\" (UniqueName: \"kubernetes.io/projected/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-kube-api-access-z5tmc\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.472945 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.472562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.490246 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.490222    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:41:45.490424 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.490286    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.490610 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.490587    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:41:45.490734 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.490608    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:41:45.490734 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.490671    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kfh6g\""
Oct 29 22:41:45.491392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491119    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.491392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491168    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmngh\""
Oct 29 22:41:45.491392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491230    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.491392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491247    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.491392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491285    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.491392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491288    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qj89s\""
Oct 29 22:41:45.491392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491383    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.491392 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491397    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:41:45.491937 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491427    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:41:45.491937 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.491883    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:41:45.492028 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.492007    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:41:45.492112 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.492057    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.492167 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.492111    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:41:45.492213 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.492192    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:41:45.492287 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.492272    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7jzk\""
Oct 29 22:41:45.500166 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500139    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:41:45.500261 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500193    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.500261 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500210    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:41:45.500261 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500223    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:41:45.500261 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500209    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gqfpm\""
Oct 29 22:41:45.500261 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500238    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500286    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pdbx4\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500290    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g5fdr\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500286    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-528cc\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500150    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500352    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500356    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500372    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500381    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:41:45.500537 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.500420    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.503497 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.503477    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:41:45.503648 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.503572    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:41:45.572904 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.572819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9lf4\" (UniqueName: \"kubernetes.io/projected/729ef181-4607-453c-ae28-080401ddc444-kube-api-access-x9lf4\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:45.572904 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.572869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg46r\" (UniqueName: \"kubernetes.io/projected/b117295a-3db5-4ec0-8858-43f561c7a075-kube-api-access-fg46r\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.572904 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.572899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-sys-fs\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.572922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-node-log\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.572943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-config\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.572964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-dbus\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.572985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-run\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-sys\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573029    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-etc-kubernetes\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-host-slash\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b117295a-3db5-4ec0-8858-43f561c7a075-host\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysconfig\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-host\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.573216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cnibin\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-cni-binary-copy\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-k8s-cni-cncf-io\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-slash\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-hosts-file\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-socket-dir-parent\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-daemon-config\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74d2\" (UniqueName: \"kubernetes.io/projected/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kube-api-access-b74d2\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-conf\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-tuned\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovn-node-metrics-cert\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573639    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-tls\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.573855 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-device-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-etc-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-netns\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4spv\" (UniqueName: \"kubernetes.io/projected/1c02209b-22cf-4ed8-aa61-429052c91a19-kube-api-access-w4spv\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-systemd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.573978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-script-lib\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk92n\" (UniqueName: \"kubernetes.io/projected/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-kube-api-access-sk92n\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-tmp\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-kubelet\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-hostroot\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-systemd-units\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-netns\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-systemd\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.574722 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-metrics-client-ca\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b29bj\" (UniqueName: \"kubernetes.io/projected/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-kube-api-access-b29bj\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-socket-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-var-lib-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-bin\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/549afed0-003b-43bf-8154-49e89fcba816-agent-certs\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-iptables-alerter-script\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tmc\" (UniqueName: \"kubernetes.io/projected/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-kube-api-access-z5tmc\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-conf-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-log-socket\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-kubelet-config\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-lib-modules\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbwb\" (UniqueName: \"kubernetes.io/projected/09138ffd-6ca5-4c22-bca1-fac58f4849c1-kube-api-access-9nbwb\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.575604 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-netd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-env-overrides\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-sys\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/549afed0-003b-43bf-8154-49e89fcba816-konnectivity-ca\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-root\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-multus\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.574995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-cnibin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-modprobe-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-textfile\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-system-cni-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-bin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-multus-certs\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b117295a-3db5-4ec0-8858-43f561c7a075-serviceca\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-tmp-dir\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-kubernetes\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-var-lib-kubelet\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-system-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctk5g\" (UniqueName: \"kubernetes.io/projected/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-kube-api-access-ctk5g\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.576434 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdc7\" (UniqueName: \"kubernetes.io/projected/ba102785-5e33-4eeb-9cc8-4f03627d328a-kube-api-access-dkdc7\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.577302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-registration-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.577302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-kubelet\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.577302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-ovn\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.577302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-wtmp\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.577302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.575523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.577302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.576280    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.577302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.576861    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-sys-fs\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.577302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.576921    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-node-log\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.577687 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.577525    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-config\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.577772 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.577752    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-dbus\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.577831 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.577817    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-run\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.577888 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.577863    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-sys\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.577943 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.577909    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-etc-kubernetes\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.578018 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.578002    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:45.578094 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.578082    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs podName:729ef181-4607-453c-ae28-080401ddc444 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:46.078054266 +0000 UTC m=+3.073518369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs") pod "network-metrics-daemon-f47cd" (UID: "729ef181-4607-453c-ae28-080401ddc444") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:45.578317 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.578299    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-host-slash\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.578397 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.578350    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b117295a-3db5-4ec0-8858-43f561c7a075-host\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.578459 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.578406    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.578513 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.578459    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysconfig\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.578570 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.578510    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-host\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.578570 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.578554    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cnibin\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.579240 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.579214    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-cni-binary-copy\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.579376 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.579284    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-k8s-cni-cncf-io\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.579376 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.579328    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-slash\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.579497 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.579375    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-hosts-file\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.579497 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.579416    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-socket-dir-parent\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.579807 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.579782    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-daemon-config\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.580062 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.580043    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.580226 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.580204    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-conf\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.580480 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.580451    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:41:45.581153 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581117    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-netd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.581265 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581164    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-system-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.581265 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581192    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-textfile\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.581265 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581223    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-log-socket\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.581437 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581267    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.581437 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581290    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-kubelet-config\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:45.581437 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581297    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-sys\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.581437 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581321    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.581437 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581326    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-system-cni-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.581437 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581366    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-bin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.581437 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581400    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-multus-certs\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.581437 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581406    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.581818 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.581744    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-kubelet\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.582068 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582040    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-registration-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.582189 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582145    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-kubelet\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.582510 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582187    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-wtmp\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.582510 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582197    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-hostroot\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.582510 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582248    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-systemd-units\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.582510 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582259    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-ovn\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.582510 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582463    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-netns\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.583006 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582521    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-env-overrides\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.583006 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582624    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.583006 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582654    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-kubernetes\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.583180 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.583014    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-tmp-dir\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.583180 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.583015    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-metrics-client-ca\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.583344 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.583274    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-iptables-alerter-script\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.583500 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.583474    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b117295a-3db5-4ec0-8858-43f561c7a075-serviceca\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.583575 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.583551    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-var-lib-kubelet\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.583637 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.583594    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-conf-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.583637 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.583594    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.583736 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.583674    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-cnibin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584366    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/549afed0-003b-43bf-8154-49e89fcba816-konnectivity-ca\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584472    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-modprobe-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584558    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-lib-modules\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.582526    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-systemd\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584613    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-var-lib-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584715    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-socket-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584769    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-device-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584892    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584939    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-systemd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.584983    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-netns\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.585022    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-tuned\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.585040    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.585096    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-bin\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.585147    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-multus\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.585235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.585209    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-root\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.586023 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.585884    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/549afed0-003b-43bf-8154-49e89fcba816-agent-certs\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:45.586023 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.585983    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-etc-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.586131 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.586068    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:45.586131 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.586123    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret podName:56ee256a-8f55-4c1d-bc82-f3cb5c3b2383 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:46.086105899 +0000 UTC m=+3.081570004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret") pod "global-pull-secret-syncer-6vdmx" (UID: "56ee256a-8f55-4c1d-bc82-f3cb5c3b2383") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:45.586265 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.586207    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.586606 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.586563    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-script-lib\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.586684 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.586639    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-tmp\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.586741 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.586709    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.586825 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.586738    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovn-node-metrics-cert\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.587125 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.587102    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.588880 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.588853    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-tls\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.597403 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.597344    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9lf4\" (UniqueName: \"kubernetes.io/projected/729ef181-4607-453c-ae28-080401ddc444-kube-api-access-x9lf4\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:45.598566 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.598541    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:45.598669 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.598570    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:45.598669 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.598584    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fvvms for pod openshift-network-diagnostics/network-check-target-j26t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:45.598669 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:45.598656    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms podName:7b8511e6-8125-4bdc-8542-aa58c941f70d nodeName:}" failed. No retries permitted until 2026-10-29 22:41:46.09863642 +0000 UTC m=+3.094100525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fvvms" (UniqueName: "kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms") pod "network-check-target-j26t7" (UID: "7b8511e6-8125-4bdc-8542-aa58c941f70d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:45.600336 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.600313    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg46r\" (UniqueName: \"kubernetes.io/projected/b117295a-3db5-4ec0-8858-43f561c7a075-kube-api-access-fg46r\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.600491 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.600473    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74d2\" (UniqueName: \"kubernetes.io/projected/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kube-api-access-b74d2\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.602773 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.602749    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctk5g\" (UniqueName: \"kubernetes.io/projected/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-kube-api-access-ctk5g\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.602843 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.602784    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbwb\" (UniqueName: \"kubernetes.io/projected/09138ffd-6ca5-4c22-bca1-fac58f4849c1-kube-api-access-9nbwb\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.605107 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.605084    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4spv\" (UniqueName: \"kubernetes.io/projected/1c02209b-22cf-4ed8-aa61-429052c91a19-kube-api-access-w4spv\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.607141 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.607116    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b29bj\" (UniqueName: \"kubernetes.io/projected/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-kube-api-access-b29bj\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.607261 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.607243    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tmc\" (UniqueName: \"kubernetes.io/projected/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-kube-api-access-z5tmc\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.608217 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.608135    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk92n\" (UniqueName: \"kubernetes.io/projected/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-kube-api-access-sk92n\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.609322 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.609303    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdc7\" (UniqueName: \"kubernetes.io/projected/ba102785-5e33-4eeb-9cc8-4f03627d328a-kube-api-access-dkdc7\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.681313 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.681278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:45.689347 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.689319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:41:45.694875 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.694836    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549afed0_003b_43bf_8154_49e89fcba816.slice/crio-552f3b80b24dae44b23399eaa6186ce2177d319ce473e76a2d2bef016fa28e45 WatchSource:0}: Error finding container 552f3b80b24dae44b23399eaa6186ce2177d319ce473e76a2d2bef016fa28e45: Status 404 returned error can't find the container with id 552f3b80b24dae44b23399eaa6186ce2177d319ce473e76a2d2bef016fa28e45
Oct 29 22:41:45.700349 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.700321    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffb5f7d_dcee_49b1_8efb_d0314fe27f5b.slice/crio-bdb72d771c6eeee48eb4b5cf36e64ba9e311abb4be7d1426e5eb2f67ea567521 WatchSource:0}: Error finding container bdb72d771c6eeee48eb4b5cf36e64ba9e311abb4be7d1426e5eb2f67ea567521: Status 404 returned error can't find the container with id bdb72d771c6eeee48eb4b5cf36e64ba9e311abb4be7d1426e5eb2f67ea567521
Oct 29 22:41:45.701450 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.701273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:41:45.708872 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.708845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9rm5r"
Oct 29 22:41:45.712582 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.712551    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb117295a_3db5_4ec0_8858_43f561c7a075.slice/crio-8e8ca6748d5adc3d30a9b74dabac5399af36885b616b2efb203011f9ab8be271 WatchSource:0}: Error finding container 8e8ca6748d5adc3d30a9b74dabac5399af36885b616b2efb203011f9ab8be271: Status 404 returned error can't find the container with id 8e8ca6748d5adc3d30a9b74dabac5399af36885b616b2efb203011f9ab8be271
Oct 29 22:41:45.719334 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.719310    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c02209b_22cf_4ed8_aa61_429052c91a19.slice/crio-67818907cca522b929168ba69fb1abe60826acea43db58be4caab91101322ee5 WatchSource:0}: Error finding container 67818907cca522b929168ba69fb1abe60826acea43db58be4caab91101322ee5: Status 404 returned error can't find the container with id 67818907cca522b929168ba69fb1abe60826acea43db58be4caab91101322ee5
Oct 29 22:41:45.721198 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.721164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:41:45.728298 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.728134    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:41:45.732654 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.732634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:41:45.732883 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.732860    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b17886d_b9f1_44b7_bcab_5fdebb8c3749.slice/crio-e9b95851b452bf3c76bad322d2c183dd56d375bc412adcf8ebe37588b17de69b WatchSource:0}: Error finding container e9b95851b452bf3c76bad322d2c183dd56d375bc412adcf8ebe37588b17de69b: Status 404 returned error can't find the container with id e9b95851b452bf3c76bad322d2c183dd56d375bc412adcf8ebe37588b17de69b
Oct 29 22:41:45.741310 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.741286    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e053f2_299d_4a89_be9a_bfaeb37e46f7.slice/crio-f80374fbf08c4a67d0348f6c07f545095d89eb928164e512aaf2b2f3aed0eedc WatchSource:0}: Error finding container f80374fbf08c4a67d0348f6c07f545095d89eb928164e512aaf2b2f3aed0eedc: Status 404 returned error can't find the container with id f80374fbf08c4a67d0348f6c07f545095d89eb928164e512aaf2b2f3aed0eedc
Oct 29 22:41:45.745148 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.745127    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bce0d5c_59af_4502_8354_4d7fc84c4dd5.slice/crio-5023999296aad709a4eed0a685e0229f88f155bbb9b84bfc3e219ea4bcf95c24 WatchSource:0}: Error finding container 5023999296aad709a4eed0a685e0229f88f155bbb9b84bfc3e219ea4bcf95c24: Status 404 returned error can't find the container with id 5023999296aad709a4eed0a685e0229f88f155bbb9b84bfc3e219ea4bcf95c24
Oct 29 22:41:45.759424 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.759376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:41:45.768328 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.768300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:41:45.775356 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.775315    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba102785_5e33_4eeb_9cc8_4f03627d328a.slice/crio-d52e223884244a3a9dea0f425d6f0c1900fde1c7cda3f382f87b6d39e7a39d1d WatchSource:0}: Error finding container d52e223884244a3a9dea0f425d6f0c1900fde1c7cda3f382f87b6d39e7a39d1d: Status 404 returned error can't find the container with id d52e223884244a3a9dea0f425d6f0c1900fde1c7cda3f382f87b6d39e7a39d1d
Oct 29 22:41:45.775356 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:45.775332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:41:45.783494 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.783461    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bb5a69_719b_48bf_a9e3_7ed728d315fa.slice/crio-ded4d753e4535c1c8a6b4122864c653e615eeaf34544cf587e9172c6c6de8946 WatchSource:0}: Error finding container ded4d753e4535c1c8a6b4122864c653e615eeaf34544cf587e9172c6c6de8946: Status 404 returned error can't find the container with id ded4d753e4535c1c8a6b4122864c653e615eeaf34544cf587e9172c6c6de8946
Oct 29 22:41:45.788565 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:41:45.788541    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09138ffd_6ca5_4c22_bca1_fac58f4849c1.slice/crio-82c5e78bdf5dcf77b6a6d4ee2db0b2d6dd4c7347bc137df44b62773013d2512b WatchSource:0}: Error finding container 82c5e78bdf5dcf77b6a6d4ee2db0b2d6dd4c7347bc137df44b62773013d2512b: Status 404 returned error can't find the container with id 82c5e78bdf5dcf77b6a6d4ee2db0b2d6dd4c7347bc137df44b62773013d2512b
Oct 29 22:41:46.079222 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.079185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:46.079407 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:46.079376    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:46.079486 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:46.079462    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs podName:729ef181-4607-453c-ae28-080401ddc444 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:47.07944235 +0000 UTC m=+4.074906440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs") pod "network-metrics-daemon-f47cd" (UID: "729ef181-4607-453c-ae28-080401ddc444") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:46.182861 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.182304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:46.182861 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.182366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:46.182861 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:46.182503    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:46.182861 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:46.182567    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret podName:56ee256a-8f55-4c1d-bc82-f3cb5c3b2383 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:47.182549114 +0000 UTC m=+4.178013218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret") pod "global-pull-secret-syncer-6vdmx" (UID: "56ee256a-8f55-4c1d-bc82-f3cb5c3b2383") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:46.183217 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:46.183001    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:46.183217 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:46.183026    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:46.183217 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:46.183040    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fvvms for pod openshift-network-diagnostics/network-check-target-j26t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:46.183217 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:46.183087    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms podName:7b8511e6-8125-4bdc-8542-aa58c941f70d nodeName:}" failed. No retries permitted until 2026-10-29 22:41:47.183072311 +0000 UTC m=+4.178536416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvvms" (UniqueName: "kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms") pod "network-check-target-j26t7" (UID: "7b8511e6-8125-4bdc-8542-aa58c941f70d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:46.416983 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.416670    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:45 +0000 UTC" deadline="2028-05-20 00:55:42.696622738 +0000 UTC"
Oct 29 22:41:46.416983 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.416890    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13634h13m56.279736779s"
Oct 29 22:41:46.442536 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.442244    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xj7f4" event={"ID":"ba102785-5e33-4eeb-9cc8-4f03627d328a","Type":"ContainerStarted","Data":"d52e223884244a3a9dea0f425d6f0c1900fde1c7cda3f382f87b6d39e7a39d1d"}
Oct 29 22:41:46.448222 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.448116    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgl57" event={"ID":"7b17886d-b9f1-44b7-bcab-5fdebb8c3749","Type":"ContainerStarted","Data":"e9b95851b452bf3c76bad322d2c183dd56d375bc412adcf8ebe37588b17de69b"}
Oct 29 22:41:46.449483 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.449456    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d55dr" event={"ID":"b117295a-3db5-4ec0-8858-43f561c7a075","Type":"ContainerStarted","Data":"8e8ca6748d5adc3d30a9b74dabac5399af36885b616b2efb203011f9ab8be271"}
Oct 29 22:41:46.451210 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.451185    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nfh24" event={"ID":"549afed0-003b-43bf-8154-49e89fcba816","Type":"ContainerStarted","Data":"552f3b80b24dae44b23399eaa6186ce2177d319ce473e76a2d2bef016fa28e45"}
Oct 29 22:41:46.456960 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.456926    2582 generic.go:356] "Generic (PLEG): container finished" podID="63db5a61472622137957294ef9d38f93" containerID="0e22bb1c9f3b43cd497b44fffb61f79ee88bf3cd6e4e94789ec0440ad0e0a258" exitCode=0
Oct 29 22:41:46.457111 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.457003    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal" event={"ID":"63db5a61472622137957294ef9d38f93","Type":"ContainerDied","Data":"0e22bb1c9f3b43cd497b44fffb61f79ee88bf3cd6e4e94789ec0440ad0e0a258"}
Oct 29 22:41:46.500649 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.500615    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-59ghk" event={"ID":"9bce0d5c-59af-4502-8354-4d7fc84c4dd5","Type":"ContainerStarted","Data":"5023999296aad709a4eed0a685e0229f88f155bbb9b84bfc3e219ea4bcf95c24"}
Oct 29 22:41:46.531460 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.531331    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx" event={"ID":"e1e053f2-299d-4a89-be9a-bfaeb37e46f7","Type":"ContainerStarted","Data":"f80374fbf08c4a67d0348f6c07f545095d89eb928164e512aaf2b2f3aed0eedc"}
Oct 29 22:41:46.538964 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.538873    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9rm5r" event={"ID":"1c02209b-22cf-4ed8-aa61-429052c91a19","Type":"ContainerStarted","Data":"67818907cca522b929168ba69fb1abe60826acea43db58be4caab91101322ee5"}
Oct 29 22:41:46.553367 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.553294    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xrt59" event={"ID":"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b","Type":"ContainerStarted","Data":"bdb72d771c6eeee48eb4b5cf36e64ba9e311abb4be7d1426e5eb2f67ea567521"}
Oct 29 22:41:46.557235 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.557099    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal" event={"ID":"67936acd58076934a9248bfd7519a6fc","Type":"ContainerStarted","Data":"d26b8c61482bc933b2faa5901cb4997cb55b4e7fe77f053b80719e77403f89f2"}
Oct 29 22:41:46.561484 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.561454    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" event={"ID":"09138ffd-6ca5-4c22-bca1-fac58f4849c1","Type":"ContainerStarted","Data":"82c5e78bdf5dcf77b6a6d4ee2db0b2d6dd4c7347bc137df44b62773013d2512b"}
Oct 29 22:41:46.573970 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:46.573941    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"ded4d753e4535c1c8a6b4122864c653e615eeaf34544cf587e9172c6c6de8946"}
Oct 29 22:41:47.103303 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.103217    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:47.109463 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.109433    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:47.109599 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.109533    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs podName:729ef181-4607-453c-ae28-080401ddc444 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:49.109509558 +0000 UTC m=+6.104973651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs") pod "network-metrics-daemon-f47cd" (UID: "729ef181-4607-453c-ae28-080401ddc444") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:47.203970 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.203736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:47.203970 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.203797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:47.203970 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.203935    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:47.204353 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.203995    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret podName:56ee256a-8f55-4c1d-bc82-f3cb5c3b2383 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:49.203976785 +0000 UTC m=+6.199440881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret") pod "global-pull-secret-syncer-6vdmx" (UID: "56ee256a-8f55-4c1d-bc82-f3cb5c3b2383") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:47.204755 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.204462    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:47.204755 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.204488    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:47.204755 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.204501    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fvvms for pod openshift-network-diagnostics/network-check-target-j26t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:47.204755 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.204549    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms podName:7b8511e6-8125-4bdc-8542-aa58c941f70d nodeName:}" failed. No retries permitted until 2026-10-29 22:41:49.204533699 +0000 UTC m=+6.199997793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvvms" (UniqueName: "kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms") pod "network-check-target-j26t7" (UID: "7b8511e6-8125-4bdc-8542-aa58c941f70d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:47.417360 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.417204    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:45 +0000 UTC" deadline="2028-07-10 05:41:09.996464658 +0000 UTC"
Oct 29 22:41:47.417360 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.417271    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14862h59m22.579198443s"
Oct 29 22:41:47.421327 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.421299    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:47.421468 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.421429    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:41:47.421854 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.421833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:47.421977 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.421953    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:41:47.422056 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.422030    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:47.422132 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:47.422115    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:41:47.594671 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.594632    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal" event={"ID":"63db5a61472622137957294ef9d38f93","Type":"ContainerStarted","Data":"e09536e8a156a0b8819db573a40431ae76897700262b514dc91438ffde95d59f"}
Oct 29 22:41:47.641027 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:47.640964    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal" podStartSLOduration=2.640944345 podStartE2EDuration="2.640944345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:46.598486028 +0000 UTC m=+3.593950141" watchObservedRunningTime="2026-10-29 22:41:47.640944345 +0000 UTC m=+4.636408462"
Oct 29 22:41:48.620347 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:48.620266    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgl57" event={"ID":"7b17886d-b9f1-44b7-bcab-5fdebb8c3749","Type":"ContainerStarted","Data":"0e12a443499a82fc81d5b2aed3a679009bf4a8aaf395369915f0775d18f02e81"}
Oct 29 22:41:48.667069 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:48.666247    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal" podStartSLOduration=3.666143429 podStartE2EDuration="3.666143429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:47.642031626 +0000 UTC m=+4.637495740" watchObservedRunningTime="2026-10-29 22:41:48.666143429 +0000 UTC m=+5.661607543"
Oct 29 22:41:49.123296 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.122579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:49.123296 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.122730    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:49.123296 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.122802    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs podName:729ef181-4607-453c-ae28-080401ddc444 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:53.122772073 +0000 UTC m=+10.118236177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs") pod "network-metrics-daemon-f47cd" (UID: "729ef181-4607-453c-ae28-080401ddc444") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:49.223547 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.223508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:49.223747 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.223567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:49.223747 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.223710    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:49.223865 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.223773    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret podName:56ee256a-8f55-4c1d-bc82-f3cb5c3b2383 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:53.223755065 +0000 UTC m=+10.219219160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret") pod "global-pull-secret-syncer-6vdmx" (UID: "56ee256a-8f55-4c1d-bc82-f3cb5c3b2383") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:49.224339 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.224315    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:49.224339 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.224343    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:49.224508 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.224356    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fvvms for pod openshift-network-diagnostics/network-check-target-j26t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:49.224508 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.224414    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms podName:7b8511e6-8125-4bdc-8542-aa58c941f70d nodeName:}" failed. No retries permitted until 2026-10-29 22:41:53.224398938 +0000 UTC m=+10.219863087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvvms" (UniqueName: "kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms") pod "network-check-target-j26t7" (UID: "7b8511e6-8125-4bdc-8542-aa58c941f70d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:49.420211 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.420133    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:49.420382 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.420279    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:41:49.420702 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.420676    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:49.420802 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.420777    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:41:49.420863 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.420845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:49.420952 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:49.420933    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:41:49.631724 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.631682    2582 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="326f05f6c3dc88c016f25dd04922aac1983484d92133e876feefa3c01c9f7110" exitCode=0
Oct 29 22:41:49.632287 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.631782    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" event={"ID":"09138ffd-6ca5-4c22-bca1-fac58f4849c1","Type":"ContainerDied","Data":"326f05f6c3dc88c016f25dd04922aac1983484d92133e876feefa3c01c9f7110"}
Oct 29 22:41:49.636451 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.636421    2582 generic.go:356] "Generic (PLEG): container finished" podID="7b17886d-b9f1-44b7-bcab-5fdebb8c3749" containerID="0e12a443499a82fc81d5b2aed3a679009bf4a8aaf395369915f0775d18f02e81" exitCode=0
Oct 29 22:41:49.636567 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:49.636470    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgl57" event={"ID":"7b17886d-b9f1-44b7-bcab-5fdebb8c3749","Type":"ContainerDied","Data":"0e12a443499a82fc81d5b2aed3a679009bf4a8aaf395369915f0775d18f02e81"}
Oct 29 22:41:50.647926 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:50.647885    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgl57" event={"ID":"7b17886d-b9f1-44b7-bcab-5fdebb8c3749","Type":"ContainerStarted","Data":"cd5dceb36aad4d36353f7e81ce083be57e8d4ce8e893c2ef56368bdf2d9b6772"}
Oct 29 22:41:50.648445 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:50.647997    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgl57" event={"ID":"7b17886d-b9f1-44b7-bcab-5fdebb8c3749","Type":"ContainerStarted","Data":"ea1c1a68eaaedec3e7c051ac83d9a826580c25cbf1764abdf7721daedbe1a94c"}
Oct 29 22:41:50.653382 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:50.652848    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nfh24" event={"ID":"549afed0-003b-43bf-8154-49e89fcba816","Type":"ContainerStarted","Data":"30ea78e27b8b7275b7d76fe28fbf4f66afb9dd788dacb49a75bedd77bd22fd6f"}
Oct 29 22:41:50.737299 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:50.736314    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nfh24" podStartSLOduration=2.613909707 podStartE2EDuration="6.736294291s" totalImagesPullingTime="4.122384584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:50.735348555 +0000 UTC m=+7.730812677" watchObservedRunningTime="2026-10-29 22:41:50.736294291 +0000 UTC m=+7.731758404"
Oct 29 22:41:50.737299 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:50.737088    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tgl57" podStartSLOduration=4.011517822 podStartE2EDuration="6.737071683s" totalImagesPullingTime="2.593241858s" totalInitContainerRuntime="132.312003ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:50.69343451 +0000 UTC m=+7.688898642" watchObservedRunningTime="2026-10-29 22:41:50.737071683 +0000 UTC m=+7.732535796"
Oct 29 22:41:51.425318 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:51.425228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:51.425493 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:51.425370    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:41:51.425777 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:51.425757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:51.425893 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:51.425870    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:41:51.425965 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:51.425944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:51.426044 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:51.426025    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:41:51.659669 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:51.659634    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d55dr" event={"ID":"b117295a-3db5-4ec0-8858-43f561c7a075","Type":"ContainerStarted","Data":"576605d6d253e19634563cbd66b56784e87974a25f5534dd5bfd2c5b6af66699"}
Oct 29 22:41:51.701806 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:51.701705    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d55dr" podStartSLOduration=2.342058454 podStartE2EDuration="7.701687557s" totalImagesPullingTime="5.359629103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:51.700998847 +0000 UTC m=+8.696462961" watchObservedRunningTime="2026-10-29 22:41:51.701687557 +0000 UTC m=+8.697151672"
Oct 29 22:41:52.663365 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:52.663321    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx" event={"ID":"e1e053f2-299d-4a89-be9a-bfaeb37e46f7","Type":"ContainerStarted","Data":"d240af6068c7a9b4185a37e4be697f0fd8690aee60fd8a8e356bbb2edad6cd6d"}
Oct 29 22:41:53.161578 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:53.161473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:53.161738 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.161652    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:53.161738 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.161717    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs podName:729ef181-4607-453c-ae28-080401ddc444 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:01.161699285 +0000 UTC m=+18.157163390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs") pod "network-metrics-daemon-f47cd" (UID: "729ef181-4607-453c-ae28-080401ddc444") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:53.262121 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:53.262059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:53.262300 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:53.262128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:53.262300 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.262254    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:53.262536 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.262311    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret podName:56ee256a-8f55-4c1d-bc82-f3cb5c3b2383 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:01.262294533 +0000 UTC m=+18.257758635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret") pod "global-pull-secret-syncer-6vdmx" (UID: "56ee256a-8f55-4c1d-bc82-f3cb5c3b2383") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:53.262738 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.262718    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:53.262827 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.262743    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:53.262827 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.262755    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fvvms for pod openshift-network-diagnostics/network-check-target-j26t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:53.262827 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.262808    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms podName:7b8511e6-8125-4bdc-8542-aa58c941f70d nodeName:}" failed. No retries permitted until 2026-10-29 22:42:01.262793173 +0000 UTC m=+18.258257279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvvms" (UniqueName: "kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms") pod "network-check-target-j26t7" (UID: "7b8511e6-8125-4bdc-8542-aa58c941f70d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:53.424500 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:53.423656    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:53.424500 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.423784    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:41:53.424500 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:53.424176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:53.424500 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.424267    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:41:53.424500 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:53.424331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:53.424500 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:53.424399    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:41:53.812936 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:53.811919    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:41:54.429879 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.429711    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:41:53.812934294Z","UUID":"1f2377be-5146-4a4c-a241-d5776cfd5160","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:41:54.431927 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.431477    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:41:54.431927 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.431511    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:41:54.674509 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.674469    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xrt59" event={"ID":"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b","Type":"ContainerStarted","Data":"b8e7b16b5ef43a57aafd75e084f1ebef2de9fd5d51f63d58ba6300faf4ec5203"}
Oct 29 22:41:54.677321 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.677287    2582 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="2629419e54b80c0cac71825c6fbb9ed2349b7a3958fdbc9772f30058764151d7" exitCode=0
Oct 29 22:41:54.677422 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.677391    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" event={"ID":"09138ffd-6ca5-4c22-bca1-fac58f4849c1","Type":"ContainerDied","Data":"2629419e54b80c0cac71825c6fbb9ed2349b7a3958fdbc9772f30058764151d7"}
Oct 29 22:41:54.681203 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.681177    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx" event={"ID":"e1e053f2-299d-4a89-be9a-bfaeb37e46f7","Type":"ContainerStarted","Data":"dcfdeb843d860d2342cb04ba0edca1266de948d28ae100a0755e6c8d12d0d04c"}
Oct 29 22:41:54.682982 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.682961    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9rm5r" event={"ID":"1c02209b-22cf-4ed8-aa61-429052c91a19","Type":"ContainerStarted","Data":"9a8575dcd1aa08c8e81aa749328c17ac771e23718d1f67e4f1a378338f11a4a0"}
Oct 29 22:41:54.756337 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.756275    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9rm5r" podStartSLOduration=2.627210736 podStartE2EDuration="10.7562575s" totalImagesPullingTime="8.129046764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:54.755909032 +0000 UTC m=+11.751373132" watchObservedRunningTime="2026-10-29 22:41:54.7562575 +0000 UTC m=+11.751721611"
Oct 29 22:41:54.756766 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:54.756737    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xrt59" podStartSLOduration=2.498142684 podStartE2EDuration="10.75672818s" totalImagesPullingTime="8.258585496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:54.714665746 +0000 UTC m=+11.710129856" watchObservedRunningTime="2026-10-29 22:41:54.75672818 +0000 UTC m=+11.752192333"
Oct 29 22:41:55.421690 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.421666    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:55.422347 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:55.421777    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:41:55.422347 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.421848    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:55.422347 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:55.421897    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:41:55.422347 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.421942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:55.422347 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:55.422000    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:41:55.583998 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.583919    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:55.584586 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.584567    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:55.688686 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.688652    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx" event={"ID":"e1e053f2-299d-4a89-be9a-bfaeb37e46f7","Type":"ContainerStarted","Data":"582392f4be11ed4e37370c4e8701405d1c6e9849bc2c0e3a0ea3ef9a7bb1aa34"}
Oct 29 22:41:55.691765 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.691483    2582 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="b42e19096340b1b87501e7f00b0fc3d57a229ca911b53656e918dd26718d74a3" exitCode=0
Oct 29 22:41:55.691765 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.691515    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" event={"ID":"09138ffd-6ca5-4c22-bca1-fac58f4849c1","Type":"ContainerDied","Data":"b42e19096340b1b87501e7f00b0fc3d57a229ca911b53656e918dd26718d74a3"}
Oct 29 22:41:55.770606 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:55.770557    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx" podStartSLOduration=3.263457041 podStartE2EDuration="11.770540564s" totalImagesPullingTime="8.507083523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:55.727265193 +0000 UTC m=+12.722729302" watchObservedRunningTime="2026-10-29 22:41:55.770540564 +0000 UTC m=+12.766004674"
Oct 29 22:41:56.695890 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:56.695273    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-59ghk" event={"ID":"9bce0d5c-59af-4502-8354-4d7fc84c4dd5","Type":"ContainerStarted","Data":"64d197d5805a7cff7c9df5efbd869d844e9ef80ee64d9179c0a2b60507ceed5b"}
Oct 29 22:41:56.697903 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:56.697879    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xj7f4" event={"ID":"ba102785-5e33-4eeb-9cc8-4f03627d328a","Type":"ContainerStarted","Data":"1787f1553434888d0bf702d91ff14f3d84a90ee662040114fcd0434cc7a9807a"}
Oct 29 22:41:56.733372 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:56.733331    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-59ghk" podStartSLOduration=4.181316975 podStartE2EDuration="12.73331765s" totalImagesPullingTime="8.552000675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:56.732930649 +0000 UTC m=+13.728394762" watchObservedRunningTime="2026-10-29 22:41:56.73331765 +0000 UTC m=+13.728781759"
Oct 29 22:41:56.773144 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:56.773098    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xj7f4" podStartSLOduration=2.156109181 podStartE2EDuration="12.773068029s" totalImagesPullingTime="10.616958848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:56.772243723 +0000 UTC m=+13.767707833" watchObservedRunningTime="2026-10-29 22:41:56.773068029 +0000 UTC m=+13.768532137"
Oct 29 22:41:57.352747 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:57.352718    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:57.352952 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:57.352831    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:41:57.353791 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:57.353757    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:41:57.419983 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:57.419961    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:57.420154 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:57.419983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:57.420154 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:57.420061    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:41:57.420414 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:57.420390    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:41:57.420473 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:57.420431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:57.420519 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:57.420489    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:41:57.703549 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:57.703471    2582 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="b3b62c8af76ddd476d11f869429696e9c50e74b3402078b95e1db368349beaf1" exitCode=0
Oct 29 22:41:57.703987 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:57.703575    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" event={"ID":"09138ffd-6ca5-4c22-bca1-fac58f4849c1","Type":"ContainerDied","Data":"b3b62c8af76ddd476d11f869429696e9c50e74b3402078b95e1db368349beaf1"}
Oct 29 22:41:59.421462 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:59.421437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:41:59.421794 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:59.421530    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:41:59.421794 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:59.421786    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:41:59.421858 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:59.421839    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:41:59.421887 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:59.421876    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:41:59.421938 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:41:59.421924    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:41:59.711770 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:59.711681    2582 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="b22b814800c273fc5342b16bf80bf1b4ce4e7579687a8d367c694aaa0db07e73" exitCode=0
Oct 29 22:41:59.711770 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:41:59.711749    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" event={"ID":"09138ffd-6ca5-4c22-bca1-fac58f4849c1","Type":"ContainerDied","Data":"b22b814800c273fc5342b16bf80bf1b4ce4e7579687a8d367c694aaa0db07e73"}
Oct 29 22:42:00.716764 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:00.716728    2582 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="1e9517e1e2d090525ee55da0a316e7a1ecffe4d2181d0327ed10b771e4416d41" exitCode=0
Oct 29 22:42:00.717183 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:00.716798    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" event={"ID":"09138ffd-6ca5-4c22-bca1-fac58f4849c1","Type":"ContainerDied","Data":"1e9517e1e2d090525ee55da0a316e7a1ecffe4d2181d0327ed10b771e4416d41"}
Oct 29 22:42:01.228596 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.228370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:01.228703 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.228508    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:01.228703 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.228666    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs podName:729ef181-4607-453c-ae28-080401ddc444 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.228651465 +0000 UTC m=+34.224115558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs") pod "network-metrics-daemon-f47cd" (UID: "729ef181-4607-453c-ae28-080401ddc444") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:01.329484 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.329446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:01.329627 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.329499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:01.329627 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.329591    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:01.329627 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.329603    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:01.329627 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.329624    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:01.329763 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.329634    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fvvms for pod openshift-network-diagnostics/network-check-target-j26t7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:01.329763 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.329647    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret podName:56ee256a-8f55-4c1d-bc82-f3cb5c3b2383 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.329628699 +0000 UTC m=+34.325092785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret") pod "global-pull-secret-syncer-6vdmx" (UID: "56ee256a-8f55-4c1d-bc82-f3cb5c3b2383") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:01.329763 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.329673    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms podName:7b8511e6-8125-4bdc-8542-aa58c941f70d nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.329660673 +0000 UTC m=+34.325124765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fvvms" (UniqueName: "kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms") pod "network-check-target-j26t7" (UID: "7b8511e6-8125-4bdc-8542-aa58c941f70d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:01.420157 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.420129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:01.420287 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.420186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:01.420287 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.420189    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:01.420347 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.420277    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:42:01.420380 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.420340    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:42:01.420432 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:01.420415    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:42:01.721350 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.721312    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" event={"ID":"09138ffd-6ca5-4c22-bca1-fac58f4849c1","Type":"ContainerStarted","Data":"c2fbb2841583adf3ea27ae99516991f35bfb0eeee4bc5873a99a53ea520a1a20"}
Oct 29 22:42:01.723927 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.723903    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"215196db62852ae78bbb5cd048344b4aa57f95793936c0a120a79a7ecfa58103"}
Oct 29 22:42:01.724039 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.723971    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"3b5ec609fd01319320b3cfe9081952156f24bf2b923f550e8e5249c02cb9fdf2"}
Oct 29 22:42:01.724039 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.724018    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"4c4642a61a14433ce2dd403e56caccd6ac4d710edba9fc1c81b2e360a7cad1b2"}
Oct 29 22:42:01.724157 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.724062    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"990a83776d892a1441f189a123ecf2f66273eebe450d1ace7635321851dfcff5"}
Oct 29 22:42:01.724157 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.724124    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"53af5c6b4fae9e0420230c683f0ce9e7a9996d5e83285d9b28b2dc5c05da5572"}
Oct 29 22:42:01.724253 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.724171    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"3bb0498cb5c0ecd3dcdabace292dd252edb035b236eef6fe7896263cc289b8a2"}
Oct 29 22:42:01.763427 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:01.763382    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qh8k5" podStartSLOduration=7.00124255 podStartE2EDuration="17.763369137s" totalImagesPullingTime="10.762126587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:01.762845241 +0000 UTC m=+18.758309351" watchObservedRunningTime="2026-10-29 22:42:01.763369137 +0000 UTC m=+18.758833245"
Oct 29 22:42:03.421441 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:03.421400    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:03.421865 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:03.421487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:03.421865 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:03.421498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:03.421865 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:03.421557    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:42:03.421865 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:03.421609    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:42:03.421865 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:03.421553    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:42:03.731956 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:03.731920    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"4f086000fc434add88f2dc226dcae36143af75d83b787767cda355e52a0b3203"}
Oct 29 22:42:05.419982 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:05.419781    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:05.420463 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:05.419813    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:05.420463 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:05.419853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:05.420463 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:05.420078    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:42:05.420463 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:05.420140    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:42:05.420463 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:05.420215    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:42:05.739870 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:05.739835    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" event={"ID":"f7bb5a69-719b-48bf-a9e3-7ed728d315fa","Type":"ContainerStarted","Data":"ef1e7e6d3c50b2ab776d4f67c9776b8d0b27e8f2ff386aa75e2a0beb8e85514d"}
Oct 29 22:42:05.740064 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:05.740038    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:05.754942 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:05.754917    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:05.825366 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:05.825320    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw" podStartSLOduration=6.710936997 podStartE2EDuration="21.825305102s" totalImagesPullingTime="15.114368105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:05.825133389 +0000 UTC m=+22.820597497" watchObservedRunningTime="2026-10-29 22:42:05.825305102 +0000 UTC m=+22.820769213"
Oct 29 22:42:06.741875 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:06.741846    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:06.742302 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:06.742284    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:06.756350 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:06.756326    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:07.420380 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.420319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:07.420380 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.420341    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:07.420558 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.420411    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:07.420558 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:07.420531    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:42:07.420639 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:07.420562    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:42:07.420639 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:07.420629    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:42:07.461775 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.461744    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j26t7"]
Oct 29 22:42:07.465019 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.464990    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vdmx"]
Oct 29 22:42:07.465715 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.465695    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f47cd"]
Oct 29 22:42:07.552256 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.552229    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:07.743390 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.743315    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:07.743852 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.743412    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:07.743852 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:07.743478    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:42:07.743852 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:07.743596    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:42:07.743949 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:07.743897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:07.743986 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:07.743967    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:42:09.420103 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:09.420066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:09.420103 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:09.420098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:09.420498 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:09.420189    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:42:09.420498 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:09.420255    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:42:09.420498 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:09.420266    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:09.420498 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:09.420353    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:42:11.420563 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:11.420538    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:11.420939 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:11.420535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:11.420939 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:11.420657    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6vdmx" podUID="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383"
Oct 29 22:42:11.420939 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:11.420679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:11.420939 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:11.420793    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j26t7" podUID="7b8511e6-8125-4bdc-8542-aa58c941f70d"
Oct 29 22:42:11.420939 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:11.420901    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f47cd" podUID="729ef181-4607-453c-ae28-080401ddc444"
Oct 29 22:42:12.762180 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.762007    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeReady"
Oct 29 22:42:12.762552 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.762266    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 22:42:12.818744 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.818723    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hzwmx"]
Oct 29 22:42:12.823230 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.823210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:12.827459 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.827440    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5vrv4"]
Oct 29 22:42:12.830490 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.830473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:12.840715 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.840697    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qxfxw"]
Oct 29 22:42:12.843256 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.843238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:12.844122 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.844102    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzwmx"]
Oct 29 22:42:12.853006 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.852984    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5vrv4"]
Oct 29 22:42:12.862844 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.862820    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxfxw"]
Oct 29 22:42:12.865633 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.865614    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:42:12.865922 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.865901    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:42:12.865997 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.865963    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:42:12.865997 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.865963    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:42:12.866121 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.866105    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:42:12.866121 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.866116    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:42:12.866242 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.866133    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:42:12.866242 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.866150    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:42:12.866330 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.866292    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vstz\""
Oct 29 22:42:12.874070 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.874051    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:42:12.874070 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.874062    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6bbfq\""
Oct 29 22:42:12.874070 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.874053    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pxqn\""
Oct 29 22:42:12.915688 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:12.915776 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-metrics-tls\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:12.915776 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915723    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-tmp-dir\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:12.915856 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxqw\" (UniqueName: \"kubernetes.io/projected/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-api-access-tlxqw\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:12.915891 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-data-volume\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:12.915951 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:12.915992 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915958    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-config-volume\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:12.915992 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstdq\" (UniqueName: \"kubernetes.io/projected/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-kube-api-access-gstdq\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:12.916074 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:12.915993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-crio-socket\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.016529 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-tmp-dir\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.016529 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slqb\" (UniqueName: \"kubernetes.io/projected/4e048e3a-9761-4519-af80-d7e4fe16ded6-kube-api-access-7slqb\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:13.016529 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxqw\" (UniqueName: \"kubernetes.io/projected/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-api-access-tlxqw\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.016529 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e048e3a-9761-4519-af80-d7e4fe16ded6-cert\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:13.016761 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-data-volume\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.016761 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.016850 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-config-volume\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.016850 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstdq\" (UniqueName: \"kubernetes.io/projected/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-kube-api-access-gstdq\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.016850 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-crio-socket\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.016850 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.017037 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-metrics-tls\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.017037 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.016994    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-crio-socket\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.017169 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.017130    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-tmp-dir\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.017169 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.017159    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-data-volume\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.017491 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.017472    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.017581 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.017494    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-config-volume\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.020364 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.020346    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.020439 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.020374    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-metrics-tls\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.035816 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.035794    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxqw\" (UniqueName: \"kubernetes.io/projected/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-api-access-tlxqw\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.035902 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.035859    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstdq\" (UniqueName: \"kubernetes.io/projected/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-kube-api-access-gstdq\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.117551 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.117513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slqb\" (UniqueName: \"kubernetes.io/projected/4e048e3a-9761-4519-af80-d7e4fe16ded6-kube-api-access-7slqb\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:13.117697 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.117579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e048e3a-9761-4519-af80-d7e4fe16ded6-cert\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:13.119774 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.119754    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e048e3a-9761-4519-af80-d7e4fe16ded6-cert\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:13.136004 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.135980    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slqb\" (UniqueName: \"kubernetes.io/projected/4e048e3a-9761-4519-af80-d7e4fe16ded6-kube-api-access-7slqb\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:13.136782 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.136763    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:13.145020 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.145005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:13.153850 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.153829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:13.421320 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.421290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:13.421320 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.421322    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:13.421538 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.421409    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:13.436231 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.436169    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:42:13.436231 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.436227    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:42:13.436390 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.436234    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:42:13.436390 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.436358    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7wdgt\""
Oct 29 22:42:13.436390 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.436374    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4f7f9\""
Oct 29 22:42:13.436520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.436398    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:42:13.436520 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.436429    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p7zhx\""
Oct 29 22:42:13.464986 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.464680    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzwmx"]
Oct 29 22:42:13.467313 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:42:13.467291    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea35e55_0ac2_48aa_8e7d_217aae2f5445.slice/crio-3d301e8afffe550c8c04e5a4367bcbdcbcab0082487d3f05016294cfb8f8630d WatchSource:0}: Error finding container 3d301e8afffe550c8c04e5a4367bcbdcbcab0082487d3f05016294cfb8f8630d: Status 404 returned error can't find the container with id 3d301e8afffe550c8c04e5a4367bcbdcbcab0082487d3f05016294cfb8f8630d
Oct 29 22:42:13.475216 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.475199    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxfxw"]
Oct 29 22:42:13.479886 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:42:13.479865    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e048e3a_9761_4519_af80_d7e4fe16ded6.slice/crio-6b7e0d7ec2e63b3e87e88b3236200ba9348e9edec1f3d12b9c8ad234456d914d WatchSource:0}: Error finding container 6b7e0d7ec2e63b3e87e88b3236200ba9348e9edec1f3d12b9c8ad234456d914d: Status 404 returned error can't find the container with id 6b7e0d7ec2e63b3e87e88b3236200ba9348e9edec1f3d12b9c8ad234456d914d
Oct 29 22:42:13.491608 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.491591    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5vrv4"]
Oct 29 22:42:13.494045 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:42:13.494024    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520fb8b2_51a7_4ecc_9a6a_21b5c608be37.slice/crio-38b5547fa626e707fe878db0d0a0734d7aea26a32fe4150cde518d221b02957f WatchSource:0}: Error finding container 38b5547fa626e707fe878db0d0a0734d7aea26a32fe4150cde518d221b02957f: Status 404 returned error can't find the container with id 38b5547fa626e707fe878db0d0a0734d7aea26a32fe4150cde518d221b02957f
Oct 29 22:42:13.756969 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.756885    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxfxw" event={"ID":"4e048e3a-9761-4519-af80-d7e4fe16ded6","Type":"ContainerStarted","Data":"6b7e0d7ec2e63b3e87e88b3236200ba9348e9edec1f3d12b9c8ad234456d914d"}
Oct 29 22:42:13.757931 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.757882    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzwmx" event={"ID":"6ea35e55-0ac2-48aa-8e7d-217aae2f5445","Type":"ContainerStarted","Data":"3d301e8afffe550c8c04e5a4367bcbdcbcab0082487d3f05016294cfb8f8630d"}
Oct 29 22:42:13.759205 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.759184    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vrv4" event={"ID":"520fb8b2-51a7-4ecc-9a6a-21b5c608be37","Type":"ContainerStarted","Data":"14190d5e88c01cc206f5b43b3a941ffef710000067ae76c37c7c0600a2a8c401"}
Oct 29 22:42:13.759359 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:13.759231    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vrv4" event={"ID":"520fb8b2-51a7-4ecc-9a6a-21b5c608be37","Type":"ContainerStarted","Data":"38b5547fa626e707fe878db0d0a0734d7aea26a32fe4150cde518d221b02957f"}
Oct 29 22:42:14.764566 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:14.764531    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vrv4" event={"ID":"520fb8b2-51a7-4ecc-9a6a-21b5c608be37","Type":"ContainerStarted","Data":"5b0dc7c9ade441164b741d1b0525836aa4067ea0884a2caebd78710e23755c1f"}
Oct 29 22:42:15.769208 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:15.769150    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzwmx" event={"ID":"6ea35e55-0ac2-48aa-8e7d-217aae2f5445","Type":"ContainerStarted","Data":"a1a368d44926676846df8b0a6aa37e6b0216180ddd21bc282bbcd4810cfbc920"}
Oct 29 22:42:15.769533 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:15.769221    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzwmx" event={"ID":"6ea35e55-0ac2-48aa-8e7d-217aae2f5445","Type":"ContainerStarted","Data":"17f51d193dc57220c6a8c03f8b98185641152a4074c407b8204adf5cead02388"}
Oct 29 22:42:16.773660 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:16.773629    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxfxw" event={"ID":"4e048e3a-9761-4519-af80-d7e4fe16ded6","Type":"ContainerStarted","Data":"bc1b10e3534a563f3cd9beb70065ffe54866c0e79ecb37787a811127744c5a3b"}
Oct 29 22:42:16.775791 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:16.775760    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vrv4" event={"ID":"520fb8b2-51a7-4ecc-9a6a-21b5c608be37","Type":"ContainerStarted","Data":"1d3e315533a120489866214f46f67818984111fbd7a8b843e30f6022677bc7b7"}
Oct 29 22:42:16.775998 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:16.775983    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:16.814799 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:16.814726    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qxfxw" podStartSLOduration=1.7489499080000002 podStartE2EDuration="4.814714284s" totalImagesPullingTime="3.065764376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.813987568 +0000 UTC m=+33.809451677" watchObservedRunningTime="2026-10-29 22:42:16.814714284 +0000 UTC m=+33.810178388"
Oct 29 22:42:16.854430 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:16.854387    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5vrv4" podStartSLOduration=1.944456852 podStartE2EDuration="4.85437535s" totalImagesPullingTime="2.909918498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.853730938 +0000 UTC m=+33.849195048" watchObservedRunningTime="2026-10-29 22:42:16.85437535 +0000 UTC m=+33.849839459"
Oct 29 22:42:16.895101 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:16.895039    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hzwmx" podStartSLOduration=2.862233465 podStartE2EDuration="4.895026647s" totalImagesPullingTime="2.032793182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.894253636 +0000 UTC m=+33.889717746" watchObservedRunningTime="2026-10-29 22:42:16.895026647 +0000 UTC m=+33.890490795"
Oct 29 22:42:17.262323 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.262283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:17.266022 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.265995    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:17.343336 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.343301    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:17.363038 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.362846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:17.363154 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.363134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:17.365676 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.365658    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:17.366158 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.366136    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:17.635512 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.635479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:17.646599 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.646566    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:17.647469 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.647446    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f47cd"]
Oct 29 22:42:17.651283 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:42:17.651248    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729ef181_4607_453c_ae28_080401ddc444.slice/crio-742793cdd141bd320f3a8248c3443e6aef2c029a7ae1291e16e62b743cf8d037 WatchSource:0}: Error finding container 742793cdd141bd320f3a8248c3443e6aef2c029a7ae1291e16e62b743cf8d037: Status 404 returned error can't find the container with id 742793cdd141bd320f3a8248c3443e6aef2c029a7ae1291e16e62b743cf8d037
Oct 29 22:42:17.779937 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.779908    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f47cd" event={"ID":"729ef181-4607-453c-ae28-080401ddc444","Type":"ContainerStarted","Data":"742793cdd141bd320f3a8248c3443e6aef2c029a7ae1291e16e62b743cf8d037"}
Oct 29 22:42:17.962077 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.962015    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6vdmx"]
Oct 29 22:42:17.963863 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:42:17.963837    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ee256a_8f55_4c1d_bc82_f3cb5c3b2383.slice/crio-69c66614e5cd167575f46bd4af6eb49e845525de34093ab73b07280824110d45 WatchSource:0}: Error finding container 69c66614e5cd167575f46bd4af6eb49e845525de34093ab73b07280824110d45: Status 404 returned error can't find the container with id 69c66614e5cd167575f46bd4af6eb49e845525de34093ab73b07280824110d45
Oct 29 22:42:17.990014 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:17.989992    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j26t7"]
Oct 29 22:42:17.992397 ip-10-0-139-130 kubenswrapper[2582]: W1029 22:42:17.992374    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8511e6_8125_4bdc_8542_aa58c941f70d.slice/crio-4eded3d326e57d07495b120d420d9ef0bb5f2ca085efd0622054679e04241c46 WatchSource:0}: Error finding container 4eded3d326e57d07495b120d420d9ef0bb5f2ca085efd0622054679e04241c46: Status 404 returned error can't find the container with id 4eded3d326e57d07495b120d420d9ef0bb5f2ca085efd0622054679e04241c46
Oct 29 22:42:18.785664 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:18.785620    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j26t7" event={"ID":"7b8511e6-8125-4bdc-8542-aa58c941f70d","Type":"ContainerStarted","Data":"4eded3d326e57d07495b120d420d9ef0bb5f2ca085efd0622054679e04241c46"}
Oct 29 22:42:18.786993 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:18.786966    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6vdmx" event={"ID":"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383","Type":"ContainerStarted","Data":"69c66614e5cd167575f46bd4af6eb49e845525de34093ab73b07280824110d45"}
Oct 29 22:42:19.204897 ip-10-0-139-130 kubenswrapper[2582]: E1029 22:42:19.204308    2582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:42:19.792759 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:19.792722    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f47cd" event={"ID":"729ef181-4607-453c-ae28-080401ddc444","Type":"ContainerStarted","Data":"37f1f41b3741f60323633636f57e06fd82ad228ca3a1a51656e76a4d576ca722"}
Oct 29 22:42:19.793244 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:19.792782    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f47cd" event={"ID":"729ef181-4607-453c-ae28-080401ddc444","Type":"ContainerStarted","Data":"45ab42533db93fc602eb8048c5e5f17838922ca3955e445f32e4d853ab363240"}
Oct 29 22:42:19.830758 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:19.830704    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f47cd" podStartSLOduration=34.60914303 podStartE2EDuration="35.830686778s" totalImagesPullingTime="1.221543748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:19.829917516 +0000 UTC m=+36.825381625" watchObservedRunningTime="2026-10-29 22:42:19.830686778 +0000 UTC m=+36.826150889"
Oct 29 22:42:21.809203 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:21.809127    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j26t7" event={"ID":"7b8511e6-8125-4bdc-8542-aa58c941f70d","Type":"ContainerStarted","Data":"cdfa62f6e371d63baf671a6116db85ab1b5d90264303bdd34c5d1e9812b433cf"}
Oct 29 22:42:21.809664 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:21.809266    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:21.848337 ip-10-0-139-130 kubenswrapper[2582]: I1029 22:42:21.848285    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j26t7" podStartSLOduration=34.424357504 podStartE2EDuration="37.848267481s" totalImagesPullingTime="3.423909977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:21.846784951 +0000 UTC m=+38.842249097" watchObservedRunningTime="2026-10-29 22:42:21.848267481 +0000 UTC m=+38.843731592"
Oct 29 22:42:22.796149 ip-10-0-139-130 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 22:42:22.814865 ip-10-0-139-130 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 22:42:22.815059 ip-10-0-139-130 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 22:42:22.815235 ip-10-0-139-130 systemd[1]: kubelet.service: Consumed 2.384s CPU time.
Oct 29 22:42:22.829633 ip-10-0-139-130 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:42:22.910499 ip-10-0-139-130 kubenswrapper[4742]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:22.910499 ip-10-0-139-130 kubenswrapper[4742]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:42:22.910499 ip-10-0-139-130 kubenswrapper[4742]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:22.910499 ip-10-0-139-130 kubenswrapper[4742]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:22.912537 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912463    4742 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.912537 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912537    4742 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912542    4742 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912545    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912554    4742 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912557    4742 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912560    4742 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912562    4742 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912565    4742 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912568    4742 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912571    4742 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912573    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912576    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912579    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912583    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912585    4742 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912588    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912591    4742 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912594    4742 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912596    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912599    4742 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.912606 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912602    4742 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912606    4742 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912611    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912614    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912618    4742 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912623    4742 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912626    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912629    4742 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912632    4742 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912634    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912637    4742 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912639    4742 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912642    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912645    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912648    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912651    4742 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912653    4742 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912656    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912658    4742 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.913121 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912661    4742 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912663    4742 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912666    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912668    4742 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912671    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912673    4742 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912676    4742 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912678    4742 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912681    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912684    4742 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912687    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912690    4742 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912693    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912696    4742 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912698    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912702    4742 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912704    4742 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912707    4742 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912710    4742 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912712    4742 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.913619 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912715    4742 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912717    4742 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912720    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912722    4742 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912725    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912728    4742 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912736    4742 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912739    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912742    4742 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912745    4742 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912748    4742 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912750    4742 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912753    4742 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912756    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912759    4742 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912762    4742 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912765    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912768    4742 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912771    4742 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.914136 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912773    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912776    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912779    4742 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912781    4742 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912784    4742 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912786    4742 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912789    4742 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912792    4742 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912794    4742 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912797    4742 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912800    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912802    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912805    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912807    4742 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912810    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912813    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912815    4742 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912818    4742 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912821    4742 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912824    4742 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.914604 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912827    4742 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912830    4742 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912833    4742 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912843    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912846    4742 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912849    4742 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.912851    4742 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913645    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913652    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913655    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913658    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913661    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913665    4742 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913668    4742 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913671    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913673    4742 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913676    4742 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913679    4742 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913682    4742 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.915102 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913684    4742 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913687    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913689    4742 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913692    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913695    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913697    4742 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913700    4742 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913703    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913705    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913708    4742 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913711    4742 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913714    4742 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913716    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913719    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913725    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913728    4742 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913731    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913733    4742 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913736    4742 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.915626 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913739    4742 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913741    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913744    4742 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913747    4742 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913749    4742 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913752    4742 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913755    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913758    4742 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913760    4742 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913763    4742 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913766    4742 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913768    4742 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913771    4742 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913774    4742 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913777    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913779    4742 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913784    4742 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913788    4742 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913791    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.916459 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913794    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913796    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913799    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913802    4742 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913805    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913807    4742 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913810    4742 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913813    4742 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913817    4742 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913820    4742 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913823    4742 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913826    4742 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913829    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913831    4742 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913834    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913836    4742 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913840    4742 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913842    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913845    4742 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913849    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.917079 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913852    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913854    4742 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913859    4742 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913862    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913866    4742 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913869    4742 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913872    4742 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913874    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913877    4742 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913880    4742 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913882    4742 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913885    4742 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913887    4742 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913890    4742 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913893    4742 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913895    4742 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913898    4742 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913900    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913903    4742 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.917587 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913906    4742 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913910    4742 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913912    4742 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913916    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913919    4742 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913921    4742 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913924    4742 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913926    4742 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913929    4742 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913931    4742 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913934    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913936    4742 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913940    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913943    4742 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913945    4742 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913948    4742 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.913951    4742 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914431    4742 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914436    4742 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914439    4742 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.918054 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914442    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914445    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914448    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914451    4742 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914454    4742 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914456    4742 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914459    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914461    4742 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914464    4742 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914467    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914469    4742 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914472    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914475    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914480    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914484    4742 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914488    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914490    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914493    4742 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914496    4742 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914498    4742 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.918594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914501    4742 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914504    4742 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914507    4742 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914509    4742 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914512    4742 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914515    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914517    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914520    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914523    4742 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914525    4742 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914528    4742 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914531    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914533    4742 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914536    4742 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914539    4742 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914541    4742 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914544    4742 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914546    4742 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914549    4742 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914552    4742 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.919218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914554    4742 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914557    4742 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914559    4742 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914562    4742 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914565    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914569    4742 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914571    4742 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914574    4742 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914577    4742 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914580    4742 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914582    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914586    4742 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914588    4742 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914591    4742 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914594    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914596    4742 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914599    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914602    4742 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914605    4742 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.919741 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914608    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914610    4742 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914614    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914616    4742 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914619    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914622    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914625    4742 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914627    4742 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914630    4742 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914633    4742 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914636    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914638    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914641    4742 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914643    4742 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914646    4742 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914648    4742 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914651    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914653    4742 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914658    4742 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.920216 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914660    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914664    4742 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914668    4742 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914671    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914674    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914677    4742 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914679    4742 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914682    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914685    4742 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914688    4742 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914690    4742 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914693    4742 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914695    4742 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914698    4742 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914700    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914704    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914706    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914709    4742 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914711    4742 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.920753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914714    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914717    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914720    4742 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914723    4742 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914725    4742 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.914728    4742 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914812    4742 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914819    4742 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914825    4742 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914831    4742 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914837    4742 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914842    4742 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914850    4742 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914856    4742 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914859    4742 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914862    4742 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914866    4742 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914870    4742 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914874    4742 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914877    4742 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914880    4742 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914883    4742 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914886    4742 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:42:22.921272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914894    4742 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914897    4742 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914901    4742 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914904    4742 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914907    4742 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914910    4742 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914914    4742 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914917    4742 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914920    4742 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914924    4742 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914927    4742 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914930    4742 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914934    4742 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914937    4742 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914940    4742 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914944    4742 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914947    4742 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914950    4742 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914953    4742 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914956    4742 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914961    4742 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914967    4742 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914971    4742 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914974    4742 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:42:22.921873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914977    4742 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914980    4742 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914984    4742 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914988    4742 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914991    4742 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914994    4742 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.914997    4742 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915000    4742 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915003    4742 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915006    4742 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915009    4742 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915012    4742 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915015    4742 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915019    4742 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915022    4742 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915025    4742 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915028    4742 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915031    4742 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915034    4742 flags.go:64] FLAG: --help="false"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915038    4742 flags.go:64] FLAG: --hostname-override="ip-10-0-139-130.ec2.internal"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915041    4742 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915044    4742 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915047    4742 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915050    4742 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:42:22.922498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915053    4742 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915056    4742 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915060    4742 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915063    4742 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915067    4742 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915072    4742 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915075    4742 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915078    4742 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915092    4742 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915096    4742 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915099    4742 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915102    4742 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915105    4742 flags.go:64] FLAG: --lock-file=""
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915108    4742 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915112    4742 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915115    4742 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915121    4742 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915124    4742 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915127    4742 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915129    4742 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915132    4742 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915136    4742 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915139    4742 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915141    4742 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915146    4742 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:42:22.923079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915149    4742 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915153    4742 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915156    4742 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915159    4742 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915162    4742 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915165    4742 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915168    4742 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915171    4742 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915174    4742 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915181    4742 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915184    4742 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915188    4742 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915192    4742 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915195    4742 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915198    4742 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915201    4742 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915204    4742 flags.go:64] FLAG: --port="10250"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915207    4742 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915210    4742 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-084c925e6068d5185"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915213    4742 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915217    4742 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915220    4742 flags.go:64] FLAG: --register-node="true"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915223    4742 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915229    4742 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915232    4742 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915235    4742 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:42:22.923805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915238    4742 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915241    4742 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915245    4742 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915247    4742 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915250    4742 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915253    4742 flags.go:64] FLAG: --runonce="false"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915256    4742 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915259    4742 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915262    4742 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915265    4742 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915268    4742 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915271    4742 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915274    4742 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915277    4742 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915280    4742 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915283    4742 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915285    4742 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915289    4742 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915295    4742 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915298    4742 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915301    4742 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915306    4742 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915309    4742 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915312    4742 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915315    4742 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:42:22.924462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915318    4742 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915321    4742 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915324    4742 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915327    4742 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915330    4742 flags.go:64] FLAG: --v="2"
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915335    4742 flags.go:64] FLAG: --version="false"
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915339    4742 flags.go:64] FLAG: --vmodule=""
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915343    4742 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.915346    4742 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916137    4742 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916161    4742 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916167    4742 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916172    4742 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916176    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916181    4742 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916186    4742 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916190    4742 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916195    4742 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916199    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916203    4742 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916208    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.925100 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916212    4742 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916216    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916225    4742 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916230    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916234    4742 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916238    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916243    4742 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916248    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916253    4742 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916261    4742 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916266    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916270    4742 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916274    4742 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916279    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916283    4742 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916292    4742 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916297    4742 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916301    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916306    4742 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916310    4742 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.925627 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916314    4742 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916320    4742 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916324    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916329    4742 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916333    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916337    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916341    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916357    4742 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916381    4742 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916386    4742 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916392    4742 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916397    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916406    4742 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916419    4742 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916442    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916447    4742 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916450    4742 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916579    4742 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916584    4742 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.926228 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916587    4742 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916590    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916593    4742 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916596    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916598    4742 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916602    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916604    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916607    4742 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916612    4742 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916616    4742 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916619    4742 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916622    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916624    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916627    4742 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916630    4742 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916633    4742 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916635    4742 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916639    4742 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916642    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.926887 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916644    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916647    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916649    4742 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916652    4742 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916655    4742 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916657    4742 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916666    4742 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916669    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916672    4742 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916675    4742 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916677    4742 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916680    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916683    4742 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916685    4742 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916688    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916690    4742 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916693    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916695    4742 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916698    4742 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916700    4742 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.927652 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916705    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916710    4742 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916712    4742 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916715    4742 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916717    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916720    4742 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916723    4742 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916725    4742 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916730    4742 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916733    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916735    4742 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916738    4742 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916743    4742 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916747    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916750    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916754    4742 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.928421 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.916792    4742 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916832    4742 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916846    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916849    4742 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916852    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916855    4742 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916858    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916860    4742 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916863    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916866    4742 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916869    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916871    4742 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916875    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916877    4742 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916880    4742 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916883    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916886    4742 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916889    4742 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916892    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916894    4742 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.928820 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916897    4742 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916900    4742 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916902    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916905    4742 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916908    4742 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916911    4742 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916913    4742 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916916    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916919    4742 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916921    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916924    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916926    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916931    4742 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916936    4742 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916939    4742 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916942    4742 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916944    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916947    4742 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916949    4742 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.929349 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916952    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916954    4742 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916957    4742 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916960    4742 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916962    4742 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916964    4742 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916967    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916970    4742 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916973    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916975    4742 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916979    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916982    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916984    4742 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916987    4742 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916990    4742 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916992    4742 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916995    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.916997    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917000    4742 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.929936 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917003    4742 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917005    4742 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917009    4742 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917012    4742 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917015    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917017    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917021    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917024    4742 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917027    4742 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917029    4742 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917032    4742 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917034    4742 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917037    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917040    4742 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917042    4742 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917045    4742 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917047    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917050    4742 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917054    4742 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917057    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.930647 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917060    4742 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917063    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917066    4742 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917072    4742 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917076    4742 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917095    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917098    4742 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917102    4742 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917104    4742 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917107    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917110    4742 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917113    4742 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917115    4742 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917118    4742 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917120    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917123    4742 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917126    4742 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917130    4742 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917135    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917139    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.931342 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917142    4742 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917146    4742 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917149    4742 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917151    4742 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917154    4742 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917156    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917159    4742 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917162    4742 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.917164    4742 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.922043    4742 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.922057    4742 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.922075    4742 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.922080    4742 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922151    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922157    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922160    4742 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922164    4742 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922167    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922169    4742 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.931817 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922172    4742 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922175    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922178    4742 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922181    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922184    4742 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922187    4742 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922190    4742 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922192    4742 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922195    4742 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922198    4742 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922200    4742 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922204    4742 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922206    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922209    4742 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922212    4742 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922215    4742 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922217    4742 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922220    4742 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922223    4742 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922226    4742 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.932428 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922229    4742 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922232    4742 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922235    4742 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922237    4742 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922240    4742 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922243    4742 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922246    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922249    4742 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922251    4742 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922254    4742 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922257    4742 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922259    4742 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922262    4742 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922265    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922267    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922270    4742 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922272    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922275    4742 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922278    4742 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922283    4742 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.933246 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922287    4742 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922290    4742 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922293    4742 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922296    4742 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922298    4742 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922301    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922304    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922307    4742 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922310    4742 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922312    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922315    4742 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922318    4742 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922321    4742 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922324    4742 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922326    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922329    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922332    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922335    4742 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922338    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922341    4742 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.933778 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922343    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922346    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922349    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922351    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922354    4742 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922357    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922359    4742 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922362    4742 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922365    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922368    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922370    4742 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922373    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922375    4742 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922378    4742 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922381    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922384    4742 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922388    4742 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922392    4742 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.934336 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922395    4742 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922398    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922401    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922404    4742 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922406    4742 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922409    4742 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922411    4742 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922414    4742 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922417    4742 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922419    4742 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922422    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922425    4742 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922431    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922434    4742 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922436    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922439    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922441    4742 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922444    4742 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922446    4742 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922449    4742 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.934772 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922452    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922454    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.922482    4742 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922617    4742 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922622    4742 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922624    4742 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922627    4742 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922630    4742 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922632    4742 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922635    4742 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922638    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922640    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922643    4742 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922646    4742 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922649    4742 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922651    4742 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.935286 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922654    4742 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922657    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922659    4742 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922662    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922665    4742 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922667    4742 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922670    4742 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922672    4742 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922676    4742 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922681    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922684    4742 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922687    4742 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922690    4742 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922692    4742 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922695    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922697    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922700    4742 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922703    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922706    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.935680 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922709    4742 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922711    4742 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922714    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922717    4742 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922719    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922722    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922724    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922727    4742 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922730    4742 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922732    4742 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922735    4742 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922737    4742 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922740    4742 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922743    4742 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922746    4742 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922748    4742 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922751    4742 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922753    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922756    4742 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922758    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.936160 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922761    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922763    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922781    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922786    4742 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922789    4742 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922792    4742 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922795    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922799    4742 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922801    4742 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922805    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922807    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922810    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922813    4742 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922816    4742 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922818    4742 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922821    4742 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922824    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922827    4742 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922829    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.936674 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922832    4742 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922835    4742 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922838    4742 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922841    4742 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922843    4742 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922846    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922848    4742 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922851    4742 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922854    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922856    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922859    4742 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922862    4742 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922866    4742 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922869    4742 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922872    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922875    4742 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922877    4742 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922881    4742 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922884    4742 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.937157 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922887    4742 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922889    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922892    4742 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922894    4742 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922897    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922900    4742 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922902    4742 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922909    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922912    4742 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922914    4742 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922917    4742 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922920    4742 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922922    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922926    4742 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922930    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922933    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.937607 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.922955    4742 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922987    4742 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922991    4742 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922994    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922997    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.922999    4742 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923002    4742 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923005    4742 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923008    4742 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923010    4742 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923013    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923016    4742 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923018    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923021    4742 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923023    4742 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923026    4742 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923029    4742 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923033    4742 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923036    4742 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.937993 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923039    4742 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923042    4742 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923045    4742 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923047    4742 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923050    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923054    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923057    4742 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923060    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923062    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923065    4742 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923067    4742 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923072    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923074    4742 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923077    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923080    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923098    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923102    4742 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923106    4742 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923108    4742 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923111    4742 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.938454 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923114    4742 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923117    4742 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923120    4742 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923122    4742 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923125    4742 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923127    4742 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923130    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923133    4742 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923135    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923139    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923141    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923144    4742 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923147    4742 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923149    4742 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923152    4742 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923155    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923157    4742 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923161    4742 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923163    4742 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923167    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.938933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923170    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923172    4742 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923175    4742 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923179    4742 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923182    4742 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923184    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923187    4742 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923190    4742 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923193    4742 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923195    4742 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923198    4742 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923201    4742 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923204    4742 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923207    4742 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923209    4742 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923212    4742 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923224    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923227    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923230    4742 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.939430 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923235    4742 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923241    4742 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923246    4742 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923250    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923253    4742 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923256    4742 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923259    4742 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923261    4742 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923264    4742 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923267    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923269    4742 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923272    4742 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923275    4742 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923277    4742 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923280    4742 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923282    4742 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923287    4742 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923290    4742 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923292    4742 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923295    4742 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.939892 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923298    4742 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923301    4742 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923303    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923306    4742 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923308    4742 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923311    4742 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923313    4742 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923316    4742 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:42:22.923318    4742 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.923369    4742 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.924357    4742 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.924409    4742 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.924588    4742 server.go:1014] "Starting client certificate rotation"
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.924730    4742 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.924785    4742 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:36:44 +0000 UTC" deadline="2028-06-28 08:57:18.976282994 +0000 UTC"
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.924817    4742 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14578h14m56.051469594s"
Oct 29 22:42:22.940378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.925054    4742 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:22.940759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.926110    4742 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:22.940759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.927749    4742 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:42:22.940759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.929594    4742 log.go:25] "Validated CRI v1 image API"
Oct 29 22:42:22.940759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.930346    4742 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:42:22.940759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.931966    4742 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9884d022-d94c-42a5-93d8-442042369ed2:/dev/nvme0n1p4 e3eadb42-9847-4e78-b331-72bc77e4e699:/dev/nvme0n1p3]
Oct 29 22:42:22.940922 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.931987    4742 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38b5547fa626e707fe878db0d0a0734d7aea26a32fe4150cde518d221b02957f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38b5547fa626e707fe878db0d0a0734d7aea26a32fe4150cde518d221b02957f/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d301e8afffe550c8c04e5a4367bcbdcbcab0082487d3f05016294cfb8f8630d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d301e8afffe550c8c04e5a4367bcbdcbcab0082487d3f05016294cfb8f8630d/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eded3d326e57d07495b120d420d9ef0bb5f2ca085efd0622054679e04241c46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eded3d326e57d07495b120d420d9ef0bb5f2ca085efd0622054679e04241c46/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5023999296aad709a4eed0a685e0229f88f155bbb9b84bfc3e219ea4bcf95c24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5023999296aad709a4eed0a685e0229f88f155bbb9b84bfc3e219ea4bcf95c24/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/552f3b80b24dae44b23399eaa6186ce2177d319ce473e76a2d2bef016fa28e45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/552f3b80b24dae44b23399eaa6186ce2177d319ce473e76a2d2bef016fa28e45/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67818907cca522b929168ba69fb1abe60826acea43db58be4caab91101322ee5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67818907cca522b929168ba69fb1abe60826acea43db58be4caab91101322ee5/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69c66614e5cd167575f46bd4af6eb49e845525de34093ab73b07280824110d45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69c66614e5cd167575f46bd4af6eb49e845525de34093ab73b07280824110d45/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b7e0d7ec2e63b3e87e88b3236200ba9348e9edec1f3d12b9c8ad234456d914d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b7e0d7ec2e63b3e87e88b3236200ba9348e9edec1f3d12b9c8ad234456d914d/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/742793cdd141bd320f3a8248c3443e6aef2c029a7ae1291e16e62b743cf8d037/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/742793cdd141bd320f3a8248c3443e6aef2c029a7ae1291e16e62b743cf8d037/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82c5e78bdf5dcf77b6a6d4ee2db0b2d6dd4c7347bc137df44b62773013d2512b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82c5e78bdf5dcf77b6a6d4ee2db0b2d6dd4c7347bc137df44b62773013d2512b/userdata/shm major:0 minor:105 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d0e760708084789f2b866adb72ceaf0c5ed24e8a749fc14429637c66bcaf20a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d0e760708084789f2b866adb72ceaf0c5ed24e8a749fc14429637c66bcaf20a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e8ca6748d5adc3d30a9b74dabac5399af36885b616b2efb203011f9ab8be271/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e8ca6748d5adc3d30a9b74dabac5399af36885b616b2efb203011f9ab8be271/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdb72d771c6eeee48eb4b5cf36e64ba9e311abb4be7d1426e5eb2f67ea567521/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdb72d771c6eeee48eb4b5cf36e64ba9e311abb4be7d1426e5eb2f67ea567521/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ded4d753e4535c1c8a6b4122864c653e615eeaf34544cf587e9172c6c6de8946/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ded4d753e4535c1c8a6b4122864c653e615eeaf34544cf587e9172c6c6de8946/userdata/shm major:0 minor:101 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9b95851b452bf3c76bad322d2c183dd56d375bc412adcf8ebe37588b17de69b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9b95851b452bf3c76bad322d2c183dd56d375bc412adcf8ebe37588b17de69b/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7f0bfff4d3bf8d4b834e70e374998fc8468c3163770df33395314b16974182e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7f0bfff4d3bf8d4b834e70e374998fc8468c3163770df33395314b16974182e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f80374fbf08c4a67d0348f6c07f545095d89eb928164e512aaf2b2f3aed0eedc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f80374fbf08c4a67d0348f6c07f545095d89eb928164e512aaf2b2f3aed0eedc/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09138ffd-6ca5-4c22-bca1-fac58f4849c1/volumes/kubernetes.io~projected/kube-api-access-9nbwb:{mountpoint:/var/lib/kubelet/pods/09138ffd-6ca5-4c22-bca1-fac58f4849c1/volumes/kubernetes.io~projected/kube-api-access-9nbwb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c02209b-22cf-4ed8-aa61-429052c91a19/volumes/kubernetes.io~projected/kube-api-access-w4spv:{mountpoint:/var/lib/kubelet/pods/1c02209b-22cf-4ed8-aa61-429052c91a19/volumes/kubernetes.io~projected/kube-api-access-w4spv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e048e3a-9761-4519-af80-d7e4fe16ded6/volumes/kubernetes.io~projected/kube-api-access-7slqb:{mountpoint:/var/lib/kubelet/pods/4e048e3a-9761-4519-af80-d7e4fe16ded6/volumes/kubernetes.io~projected/kube-api-access-7slqb major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e048e3a-9761-4519-af80-d7e4fe16ded6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4e048e3a-9761-4519-af80-d7e4fe16ded6/volumes/kubernetes.io~secret/cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/520fb8b2-51a7-4ecc-9a6a-21b5c608be37/volumes/kubernetes.io~projected/kube-api-access-tlxqw:{mountpoint:/var/lib/kubelet/pods/520fb8b2-51a7-4ecc-9a6a-21b5c608be37/volumes/kubernetes.io~projected/kube-api-access-tlxqw major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/520fb8b2-51a7-4ecc-9a6a-21b5c608be37/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/520fb8b2-51a7-4ecc-9a6a-21b5c608be37/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/549afed0-003b-43bf-8154-49e89fcba816/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/549afed0-003b-43bf-8154-49e89fcba816/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ea35e55-0ac2-48aa-8e7d-217aae2f5445/volumes/kubernetes.io~projected/kube-api-access-gstdq:{mountpoint:/var/lib/kubelet/pods/6ea35e55-0ac2-48aa-8e7d-217aae2f5445/volumes/kubernetes.io~projected/kube-api-access-gstdq major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ea35e55-0ac2-48aa-8e7d-217aae2f5445/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/6ea35e55-0ac2-48aa-8e7d-217aae2f5445/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/729ef181-4607-453c-ae28-080401ddc444/volumes/kubernetes.io~projected/kube-api-access-x9lf4:{mountpoint:/var/lib/kubelet/pods/729ef181-4607-453c-ae28-080401ddc444/volumes/kubernetes.io~projected/kube-api-access-x9lf4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/729ef181-4607-453c-ae28-080401ddc444/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/729ef181-4607-453c-ae28-080401ddc444/volumes/kubernetes.io~secret/metrics-certs major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~projected/kube-api-access-b29bj:{mountpoint:/var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~projected/kube-api-access-b29bj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8511e6-8125-4bdc-8542-aa58c941f70d/volumes/kubernetes.io~projected/kube-api-access-fvvms:{mountpoint:/var/lib/kubelet/pods/7b8511e6-8125-4bdc-8542-aa58c941f70d/volumes/kubernetes.io~projected/kube-api-access-fvvms major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bce0d5c-59af-4502-8354-4d7fc84c4dd5/volumes/kubernetes.io~projected/kube-api-access-z5tmc:{mountpoint:/var/lib/kubelet/pods/9bce0d5c-59af-4502-8354-4d7fc84c4dd5/volumes/kubernetes.io~projected/kube-api-access-z5tmc major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b117295a-3db5-4ec0-8858-43f561c7a075/volumes/kubernetes.io~projected/kube-api-access-fg46r:{mountpoint:/var/lib/kubelet/pods/b117295a-3db5-4ec0-8858-43f561c7a075/volumes/kubernetes.io~projected/kube-api-access-fg46r major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~projected/kube-api-access-dkdc7:{mountpoint:/var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~projected/kube-api-access-dkdc7 major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b/volumes/kubernetes.io~projected/kube-api-access-sk92n:{mountpoint:/var/lib/kubelet/pods/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b/volumes/kubernetes.io~projected/kube-api-access-sk92n major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1e053f2-299d-4a89-be9a-bfaeb37e46f7/volumes/kubernetes.io~projected/kube-api-access-b74d2:{mountpoint:/var/lib/kubelet/pods/e1e053f2-299d-4a89-be9a-bfaeb37e46f7/volumes/kubernetes.io~projected/kube-api-access-b74d2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volumes/kubernetes.io~projected/kube-api-access-ctk5g:{mountpoint:/var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volumes/kubernetes.io~projected/kube-api-access-ctk5g major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/831e8dc30ae809869b11b0ba42548c1eefb6b603c8331eeec83186e74ba20f95/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/63931191935f84b6c273e49750013f7fc166747fdad65ec92ace9c911cf5b088/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9126741166cb69ab3e072a724e58f06994ae3f6e88e2559ec60351a6b14987aa/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/f733cc3745dc0230d391256f73474d5e2587d072b6308c4b0bd3c639a4306616/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/e2a8f8e7a52b51b64e7f1726a12fe11f330cdb3d809a546cbd7e8dfc0db5287a/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/77a277b5df89bcb02d7bbdda4e176eedc2d8f8a214772c98d31ef95ffa112af6/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/containers/storage/overlay/060d013ac7ecdc8e057db46656fbf3696f1dafe5ba7f61ebedc2b2439d92303d/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/3cd340d8a782dec7a4100206d184928dfb06ae39062f94ec1ba882b41dde8c0c/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/89175357ba9ff56d5262183258ec0c4a4ddf2f27c894c2e9c2d44e2dd0f30650/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/85996a10af97574f372b4cf9d81724de544c8bbcdfbe342f01d9e48b6b875699/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/6c9b53b6173ebcb2b6dec8d62b9a87cafdaee1972b641431027144847a2310bd/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/4f05be9e0b1f62e4a302dbbd7eddc458906d3f06090cd2f9e93dfaa278149bf8/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/1788aad7e0b803c751aa3c3ae8ce8c07a0fae73d30fd13f2f7c6478398872e18/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/7334ecddc0e0f8e28cf3c9818a61f7fb218149d9582050cd392ef46a26332492/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/0cab7dfb9d46d9d1d606e3cd87638b006ed7c03e71ee47dbfc8ca484127181b7/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/2a99081777e60d2f90255c944139c706b5ab21819808dce1d69aef2627d090e3/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/3531d673bfc0b8366f8d7892a5d52a1b602da9f13a9a6ee89ce4fdc5856c526d/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/929d49a5fefca2ca6a0368453d15a625eebf1d1d1d4a9affdf6eb8dfb2ca36a0/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/1547bd70c0c2c148d5bbbbb83ad73c3e6bdcf1dbd537d0d33fb5da804b1f98f0/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/d5d95e015e182774d6109dd612dcc1303c8900064bb84ab2477209edf90ae6f4/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-194:{mountpoint:/var/lib/containers/storage/overlay/c188571d9c6988baba88d3d75411883bb7fe6baf9f7e913195e9277546388b2b/merged major:0 minor:194 fsType:overlay blockSize:0} overlay_0-199:{mountpoint:/var/lib/containers/storage/overlay/d71f5ae83fa617bd966708cb58e987f9536d3ef2e507719763afb3696c7645c2/merged major:0 minor:199 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/db0d92d94e9dd56c987cd307b4bafd908db7fdda80e433c6e4e26a9041883fbe/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/0ef392b964afed67e9abab81104b6bcc93baad29f9875a04fcf197b6db0a2c55/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/009aa1792efe0ac12c1e7b096c5c29c8bbe42c94852ed6631636df4382847b2d/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/5cd88fdb4eaa4f900adfae8c6218ef4d61385c37a183839b31112fd25bfef238/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/e0d2582176d00fdc6427dd5656804b558cbad3432c7ccb6602c08f4c92440ac4/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/0034de2e14963cd34838203c0ed3f5e1c83f89badf3e6977761921f2ea373c18/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/8869a8ef5f22c290bfcd32e91ce74391af90b71ec26bce43bd7e8450a5f57c10/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/3ca4383eaca331d65d5f34df81647be5bc4ca2d0e71c7bc0a0020d9a8a80b08c/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/eb753a1e2a08daf2d5fca60db1a4a69d51b6d2fa5369c462f68ec37347031574/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/c79e4c2eb8a8b1927ad8e9e60594888c23b415cc3fe6378e3fa9986ce0620e57/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/6ef19dc1ee5cb8a8d79c5efbcadc44325a3e415692e19e6d6a9816a490bb6a44/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/836c93214cc6d4f138d80e4daee8012b4ab9d5068cf6c1998e8b571ebf29e0c4/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/71bfaba92e4245e680b363eeea22e0d5c5888f1243ce45b4d533bf6641157bf3/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/67faace244c60185b2ca77e702e07d2dc0389ae538db00c1f0e751fe93fd6e94/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/2d9869ffe9ca804dedf0391723619e28b2dca5ebd7a27ccd2365edbd1d5d25e4/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/f53d894f5523574cc87f82a11c79c64d8c5a6d2c71df788e4890bf5c11f44621/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/3fd0b5c2d0e3ed13acb16d60a8f5aa2039e22123fcc684e4f5c1e1ef894550e8/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/98d66cfdb24edabd16092c43d993e8d32cb3dbca24df95fa20624cf23074ccaf/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/ae99385a8b5df607e22fd216fa49cd0ed94c2b578ce17f18683edaebe97d735b/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/71db11dd8cf7a0729bbf10273ff3f94be0e8021fd0d0e6fc53de3f4ed5f8bebf/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/ded0331e392d5670d14c29e761aeda2ee8dd7cd850abfdcc66af95bd238ce4c3/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/570f19f9097af7a2c3aa2af8e43b8020f6ccb5cb7b3038441975b556c7e7fe15/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/2b5acc36eca44882785fa9562c9c52350feb4e6404974257c1f1d70312c62edd/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/175b1c5355ec3d13433264bc111d1bc3d587001e836b0597f0ca1a55d9e44dc7/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/230e348bba4a3faa2bd48a35d4b71cc45ef7178a95a55551a9ec12463074f777/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/64d73bade0357ebda127c22792fe580ad29618079814ad1201ca38938d8d8dee/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/b34d17327b5c0a9c8e983de049ce30d17a704bdb08d1c0656388f6a2cc6894b6/merged major:0 minor:97 fsType:overlay blockSize:0} overlay_0-99:{mountpoint:/var/lib/containers/storage/overlay/47cc91ddb2124d0327fcc477818cdf1b491ba903d4fa99f55a8b549eba9ef0db/merged major:0 minor:99 fsType:overlay blockSize:0}]
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.939454    4742 manager.go:217] Machine: {Timestamp:2026-10-29 22:42:22.938973104 +0000 UTC m=+0.079714679 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103510 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2d3dc445d683e9cd3a394b39d188 SystemUUID:ec2b2d3d-c445-d683-e9cd-3a394b39d188 BootID:ac7770c5-61d6-49e4-b45c-9831064b2b70 Filesystems:[{Device:/var/lib/kubelet/pods/09138ffd-6ca5-4c22-bca1-fac58f4849c1/volumes/kubernetes.io~projected/kube-api-access-9nbwb DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volumes/kubernetes.io~projected/kube-api-access-ctk5g DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~projected/kube-api-access-b29bj DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1e053f2-299d-4a89-be9a-bfaeb37e46f7/volumes/kubernetes.io~projected/kube-api-access-b74d2 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f80374fbf08c4a67d0348f6c07f545095d89eb928164e512aaf2b2f3aed0eedc/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7f0bfff4d3bf8d4b834e70e374998fc8468c3163770df33395314b16974182e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-126 DeviceMajor:0 DeviceMinor:126 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e048e3a-9761-4519-af80-d7e4fe16ded6/volumes/kubernetes.io~projected/kube-api-access-7slqb DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8511e6-8125-4bdc-8542-aa58c941f70d/volumes/kubernetes.io~projected/kube-api-access-fvvms DeviceMajor:0 DeviceMinor:271 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69c66614e5cd167575f46bd4af6eb49e845525de34093ab73b07280824110d45/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eded3d326e57d07495b120d420d9ef0bb5f2ca085efd0622054679e04241c46/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ea35e55-0ac2-48aa-8e7d-217aae2f5445/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d301e8afffe550c8c04e5a4367bcbdcbcab0082487d3f05016294cfb8f8630d/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/549afed0-003b-43bf-8154-49e89fcba816/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1c02209b-22cf-4ed8-aa61-429052c91a19/volumes/kubernetes.io~projected/kube-api-access-w4spv DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdb72d771c6eeee48eb4b5cf36e64ba9e311abb4be7d1426e5eb2f67ea567521/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/b117295a-3db5-4ec0-8858-43f561c7a075/volumes/kubernetes.io~projected/kube-api-access-fg46r DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~projected/kube-api-access-dkdc7 DeviceMajor:0 DeviceMinor:70 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b7e0d7ec2e63b3e87e88b3236200ba9348e9edec1f3d12b9c8ad234456d914d/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba102785-5e33-4eeb-9cc8-4f03627d328a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/729ef181-4607-453c-ae28-080401ddc444/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:260 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b/volumes/kubernetes.io~projected/kube-api-access-sk92n DeviceMajor:0 DeviceMinor:69 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38b5547fa626e707fe878db0d0a0734d7aea26a32fe4150cde518d221b02957f/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e8ca6748d5adc3d30a9b74dabac5399af36885b616b2efb203011f9ab8be271/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ded4d753e4535c1c8a6b4122864c653e615eeaf34544cf587e9172c6c6de8946/userdata/shm DeviceMajor:0 DeviceMinor:101 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/520fb8b2-51a7-4ecc-9a6a-21b5c608be37/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/742793cdd141bd320f3a8248c3443e6aef2c029a7ae1291e16e62b743cf8d037/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/729ef181-4607-453c-ae28-080401ddc444/volumes/kubernetes.io~projected/kube-api-access-x9lf4 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9bce0d5c-59af-4502-8354-4d7fc84c4dd5/volumes/kubernetes.io~projected/kube-api-access-z5tmc DeviceMajor:0 DeviceMinor:68 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5023999296aad709a4eed0a685e0229f88f155bbb9b84bfc3e219ea4bcf95c24/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-99 DeviceMajor:0 DeviceMinor:99 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/520fb8b2-51a7-4ecc-9a6a-21b5c608be37/volumes/kubernetes.io~projected/kube-api-access-tlxqw DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7b17886d-b9f1-44b7-bcab-5fdebb8c3749/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82c5e78bdf5dcf77b6a6d4ee2db0b2d6dd4c7347bc137df44b62773013d2512b/userdata/shm DeviceMajor:0 DeviceMinor:105 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/552f3b80b24dae44b23399eaa6186ce2177d319ce473e76a2d2bef016fa28e45/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67818907cca522b929168ba69fb1abe60826acea43db58be4caab91101322ee5/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9b95851b452bf3c76bad322d2c183dd56d375bc412adcf8ebe37588b17de69b/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-194 DeviceMajor:0 DeviceMinor:194 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-199 DeviceMajor:0 DeviceMinor:199 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7bb5a69-719b-48bf-a9e3-7ed728d315fa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6ea35e55-0ac2-48aa-8e7d-217aae2f5445/volumes/kubernetes.io~projected/kube-api-access-gstdq DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4e048e3a-9761-4519-af80-d7e4fe16ded6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d0e760708084789f2b866adb72ceaf0c5ed24e8a749fc14429637c66bcaf20a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:38b5547fa626e70 MacAddress:c6:b8:e6:f9:1f:53 Speed:10000 Mtu:8901} {Name:3d301e8afffe550 MacAddress:ba:26:67:57:d3:2e Speed:10000 Mtu:8901} {Name:4eded3d326e57d0 MacAddress:a6:7f:78:47:ba:ca Speed:10000 Mtu:8901} {Name:69c66614e5cd167 MacAddress:02:82:49:fa:ed:40 Speed:10000 Mtu:8901} {Name:6b7e0d7ec2e63b3 MacAddress:86:eb:ed:2b:8c:5c Speed:10000 Mtu:8901} {Name:742793cdd141bd3 MacAddress:66:a9:19:d4:52:34 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c3:47:a1:c9 Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:51:2c:86:ee:f2 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:c3:47:a1:c9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:aa:32:7b:58:e6:26 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:86:bd:87:be:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.939710    4742 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.939806    4742 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940097    4742 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940127    4742 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940288    4742 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940295    4742 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940306    4742 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940325    4742 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940391    4742 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940543    4742 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940660    4742 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940693    4742 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940706    4742 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940721    4742 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940730    4742 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:42:22.941115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.940738    4742 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:42:22.941845 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.941430    4742 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:42:22.941845 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.941441    4742 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:22.942003 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.941976    4742 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:42:22.944234 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944221    4742 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:42:22.944234 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944233    4742 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:22.944352 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944343    4742 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 22:42:22.944529 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944519    4742 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:42:22.944558 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944547    4742 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:42:22.944670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944661    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:42:22.944700 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944678    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:42:22.944700 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944697    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:42:22.944753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944703    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:42:22.944753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944709    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:42:22.944753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944714    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:42:22.944753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944722    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:42:22.944753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944727    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:42:22.944753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944735    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:42:22.944753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944740    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:42:22.944753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944754    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:42:22.944971 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944764    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:42:22.944971 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944790    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:42:22.944971 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.944795    4742 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:42:22.947997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.947985    4742 server.go:1263] "Started kubelet"
Oct 29 22:42:22.949393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.948373    4742 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:42:22.949393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.948654    4742 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:42:22.949393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.948080    4742 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:42:22.949393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.949236    4742 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:42:22.948823 ip-10-0-139-130 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.950873    4742 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.952541    4742 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.952761    4742 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.952780    4742 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.952813    4742 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.952877    4742 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:45 +0000 UTC" deadline="2028-07-18 11:28:28.383161472 +0000 UTC"
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.952915    4742 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15060h46m5.430258s"
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.952923    4742 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:42:22.954572 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:22.954037    4742 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-130.ec2.internal\" not found"
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955359    4742 factory.go:55] Registering systemd factory
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955383    4742 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955496    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba102785-5e33-4eeb-9cc8-4f03627d328a" volumeName="kubernetes.io/projected/ba102785-5e33-4eeb-9cc8-4f03627d328a-kube-api-access-dkdc7" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955541    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bb5a69-719b-48bf-a9e3-7ed728d315fa" volumeName="kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-config" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955554    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bb5a69-719b-48bf-a9e3-7ed728d315fa" volumeName="kubernetes.io/projected/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-kube-api-access-ctk5g" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955565    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09138ffd-6ca5-4c22-bca1-fac58f4849c1" volumeName="kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955574    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="520fb8b2-51a7-4ecc-9a6a-21b5c608be37" volumeName="kubernetes.io/projected/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-api-access-tlxqw" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955585    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="520fb8b2-51a7-4ecc-9a6a-21b5c608be37" volumeName="kubernetes.io/secret/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955594    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549afed0-003b-43bf-8154-49e89fcba816" volumeName="kubernetes.io/secret/549afed0-003b-43bf-8154-49e89fcba816-agent-certs" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955603    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b17886d-b9f1-44b7-bcab-5fdebb8c3749" volumeName="kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955615    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b17886d-b9f1-44b7-bcab-5fdebb8c3749" volumeName="kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-tls" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955623    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba102785-5e33-4eeb-9cc8-4f03627d328a" volumeName="kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-tuned" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955633    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09138ffd-6ca5-4c22-bca1-fac58f4849c1" volumeName="kubernetes.io/projected/09138ffd-6ca5-4c22-bca1-fac58f4849c1-kube-api-access-9nbwb" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955644    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c02209b-22cf-4ed8-aa61-429052c91a19" volumeName="kubernetes.io/projected/1c02209b-22cf-4ed8-aa61-429052c91a19-kube-api-access-w4spv" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955653    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea35e55-0ac2-48aa-8e7d-217aae2f5445" volumeName="kubernetes.io/configmap/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-config-volume" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955662    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="729ef181-4607-453c-ae28-080401ddc444" volumeName="kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955673    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b17886d-b9f1-44b7-bcab-5fdebb8c3749" volumeName="kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955682    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b17886d-b9f1-44b7-bcab-5fdebb8c3749" volumeName="kubernetes.io/empty-dir/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-textfile" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955693    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b17886d-b9f1-44b7-bcab-5fdebb8c3749" volumeName="kubernetes.io/projected/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-kube-api-access-b29bj" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955702    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1e053f2-299d-4a89-be9a-bfaeb37e46f7" volumeName="kubernetes.io/projected/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kube-api-access-b74d2" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955710    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e048e3a-9761-4519-af80-d7e4fe16ded6" volumeName="kubernetes.io/projected/4e048e3a-9761-4519-af80-d7e4fe16ded6-kube-api-access-7slqb" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955719    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="520fb8b2-51a7-4ecc-9a6a-21b5c608be37" volumeName="kubernetes.io/configmap/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955728    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba102785-5e33-4eeb-9cc8-4f03627d328a" volumeName="kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-tmp" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955738    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bb5a69-719b-48bf-a9e3-7ed728d315fa" volumeName="kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-env-overrides" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955748    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bb5a69-719b-48bf-a9e3-7ed728d315fa" volumeName="kubernetes.io/secret/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955756    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e048e3a-9761-4519-af80-d7e4fe16ded6" volumeName="kubernetes.io/secret/4e048e3a-9761-4519-af80-d7e4fe16ded6-cert" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955764    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="520fb8b2-51a7-4ecc-9a6a-21b5c608be37" volumeName="kubernetes.io/empty-dir/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-data-volume" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955773    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549afed0-003b-43bf-8154-49e89fcba816" volumeName="kubernetes.io/configmap/549afed0-003b-43bf-8154-49e89fcba816-konnectivity-ca" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955781    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bffb5f7d-dcee-49b1-8efb-d0314fe27f5b" volumeName="kubernetes.io/empty-dir/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-tmp-dir" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955790    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bb5a69-719b-48bf-a9e3-7ed728d315fa" volumeName="kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955803    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="729ef181-4607-453c-ae28-080401ddc444" volumeName="kubernetes.io/projected/729ef181-4607-453c-ae28-080401ddc444-kube-api-access-x9lf4" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955813    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b17886d-b9f1-44b7-bcab-5fdebb8c3749" volumeName="kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-metrics-client-ca" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955822    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b117295a-3db5-4ec0-8858-43f561c7a075" volumeName="kubernetes.io/configmap/b117295a-3db5-4ec0-8858-43f561c7a075-serviceca" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955834    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bffb5f7d-dcee-49b1-8efb-d0314fe27f5b" volumeName="kubernetes.io/projected/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-kube-api-access-sk92n" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955843    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b117295a-3db5-4ec0-8858-43f561c7a075" volumeName="kubernetes.io/projected/b117295a-3db5-4ec0-8858-43f561c7a075-kube-api-access-fg46r" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955852    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09138ffd-6ca5-4c22-bca1-fac58f4849c1" volumeName="kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955861    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09138ffd-6ca5-4c22-bca1-fac58f4849c1" volumeName="kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955872    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56ee256a-8f55-4c1d-bc82-f3cb5c3b2383" volumeName="kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955882    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea35e55-0ac2-48aa-8e7d-217aae2f5445" volumeName="kubernetes.io/empty-dir/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-tmp-dir" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955890    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea35e55-0ac2-48aa-8e7d-217aae2f5445" volumeName="kubernetes.io/projected/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-kube-api-access-gstdq" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955899    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea35e55-0ac2-48aa-8e7d-217aae2f5445" volumeName="kubernetes.io/secret/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-metrics-tls" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955908    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8511e6-8125-4bdc-8542-aa58c941f70d" volumeName="kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955917    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bce0d5c-59af-4502-8354-4d7fc84c4dd5" volumeName="kubernetes.io/configmap/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-iptables-alerter-script" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955930    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c02209b-22cf-4ed8-aa61-429052c91a19" volumeName="kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955943    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c02209b-22cf-4ed8-aa61-429052c91a19" volumeName="kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-daemon-config" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955975    4742 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bce0d5c-59af-4502-8354-4d7fc84c4dd5" volumeName="kubernetes.io/projected/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-kube-api-access-z5tmc" seLinuxMountContext=""
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.955995    4742 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.956005    4742 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.957239    4742 factory.go:153] Registering CRI-O factory
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.957258    4742 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.957335    4742 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.957367    4742 factory.go:103] Registering Raw factory
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.957387    4742 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:42:22.958110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.957877    4742 manager.go:319] Starting recovery of all containers
Oct 29 22:42:22.981320 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.981253    4742 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:42:22.983235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.983213    4742 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:42:22.983235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.983236    4742 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:42:22.983445 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.983254    4742 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:42:22.983445 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:22.983291    4742 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 22:42:22.997231 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.997200    4742 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 22:42:22.997773 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:22.997753    4742 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:23.005685 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.005654    4742 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="1e9517e1e2d090525ee55da0a316e7a1ecffe4d2181d0327ed10b771e4416d41" exitCode=0
Oct 29 22:42:23.005685 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.005688    4742 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="b22b814800c273fc5342b16bf80bf1b4ce4e7579687a8d367c694aaa0db07e73" exitCode=0
Oct 29 22:42:23.005831 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.005696    4742 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="b3b62c8af76ddd476d11f869429696e9c50e74b3402078b95e1db368349beaf1" exitCode=0
Oct 29 22:42:23.005831 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.005704    4742 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="b42e19096340b1b87501e7f00b0fc3d57a229ca911b53656e918dd26718d74a3" exitCode=0
Oct 29 22:42:23.005831 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.005709    4742 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="2629419e54b80c0cac71825c6fbb9ed2349b7a3958fdbc9772f30058764151d7" exitCode=0
Oct 29 22:42:23.005831 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.005714    4742 generic.go:356] "Generic (PLEG): container finished" podID="09138ffd-6ca5-4c22-bca1-fac58f4849c1" containerID="326f05f6c3dc88c016f25dd04922aac1983484d92133e876feefa3c01c9f7110" exitCode=0
Oct 29 22:42:23.009224 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.009209    4742 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:23.009955 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.009928    4742 generic.go:356] "Generic (PLEG): container finished" podID="7b17886d-b9f1-44b7-bcab-5fdebb8c3749" containerID="0e12a443499a82fc81d5b2aed3a679009bf4a8aaf395369915f0775d18f02e81" exitCode=0
Oct 29 22:42:23.010035 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.010002    4742 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:23.020046 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.020018    4742 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:23.020537 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.020517    4742 generic.go:356] "Generic (PLEG): container finished" podID="63db5a61472622137957294ef9d38f93" containerID="0e22bb1c9f3b43cd497b44fffb61f79ee88bf3cd6e4e94789ec0440ad0e0a258" exitCode=0
Oct 29 22:42:23.038171 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.037930    4742 manager.go:324] Recovery completed
Oct 29 22:42:23.055758 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055706    4742 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:42:23.055758 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055720    4742 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:42:23.055758 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055734    4742 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:42:23.055893 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055846    4742 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 22:42:23.055893 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055856    4742 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 22:42:23.055893 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055873    4742 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 22:42:23.055893 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055878    4742 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 22:42:23.055893 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055883    4742 policy_none.go:50] "Start"
Oct 29 22:42:23.055893 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055891    4742 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:42:23.056065 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.055899    4742 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:42:23.056065 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.056015    4742 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 22:42:23.056065 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.056021    4742 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 22:42:23.056065 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.056026    4742 policy_none.go:44] "Start"
Oct 29 22:42:23.060496 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.060471    4742 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:42:23.063099 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.063074    4742 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:42:23.063568 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:23.063113    4742 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:42:23.063640 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.063570    4742 server.go:87] "Starting device plugin registration server"
Oct 29 22:42:23.063841 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.063827    4742 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:42:23.063900 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.063843    4742 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:42:23.064020 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.063947    4742 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:42:23.064216 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.064140    4742 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:42:23.064216 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.064198    4742 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:42:23.064216 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.064206    4742 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:42:23.064369 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.064247    4742 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:23.064184443Z","UUID":"2551275d-c464-4408-a693-a99859e1055b","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:42:23.066163 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:23.065788    4742 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:42:23.066290 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.066234    4742 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:42:23.066290 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.066256    4742 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:42:23.083392 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.083346    4742 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"]
Oct 29 22:42:23.083975 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.083959    4742 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.084032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.084000    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal" event={"ID":"63db5a61472622137957294ef9d38f93","Type":"ContainerStarted","Data":"e09536e8a156a0b8819db573a40431ae76897700262b514dc91438ffde95d59f"}
Oct 29 22:42:23.084063 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.084047    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal" event={"ID":"63db5a61472622137957294ef9d38f93","Type":"ContainerDied","Data":"0e22bb1c9f3b43cd497b44fffb61f79ee88bf3cd6e4e94789ec0440ad0e0a258"}
Oct 29 22:42:23.084119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.084069    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal" event={"ID":"63db5a61472622137957294ef9d38f93","Type":"ContainerStarted","Data":"8d0e760708084789f2b866adb72ceaf0c5ed24e8a749fc14429637c66bcaf20a"}
Oct 29 22:42:23.084160 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.084116    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal" event={"ID":"67936acd58076934a9248bfd7519a6fc","Type":"ContainerStarted","Data":"d26b8c61482bc933b2faa5901cb4997cb55b4e7fe77f053b80719e77403f89f2"}
Oct 29 22:42:23.084160 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.084134    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal" event={"ID":"67936acd58076934a9248bfd7519a6fc","Type":"ContainerStarted","Data":"f7f0bfff4d3bf8d4b834e70e374998fc8468c3163770df33395314b16974182e"}
Oct 29 22:42:23.084297 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.084285    4742 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.110377 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.110361    4742 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:23.110444 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:23.110400    4742 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-130.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.110444 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.110362    4742 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:23.110517 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:23.110446    4742 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.164133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.164113    4742 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:23.165265 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.165244    4742 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:23.165346 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.165273    4742 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:23.165346 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.165287    4742 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-130.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:23.165346 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.165319    4742 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.198515 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.198481    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67936acd58076934a9248bfd7519a6fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-130.ec2.internal\" (UID: \"67936acd58076934a9248bfd7519a6fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.198515 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.198516    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.198639 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.198534    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.202434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.202419    4742 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.202579 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.202567    4742 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.299005 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.298973    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67936acd58076934a9248bfd7519a6fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-130.ec2.internal\" (UID: \"67936acd58076934a9248bfd7519a6fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.299155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.299016    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.299155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.299038    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.299155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.299078    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.299155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.299108    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67936acd58076934a9248bfd7519a6fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-130.ec2.internal\" (UID: \"67936acd58076934a9248bfd7519a6fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.299155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.299108    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63db5a61472622137957294ef9d38f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" (UID: \"63db5a61472622137957294ef9d38f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:23.941837 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.941804    4742 apiserver.go:52] "Watching apiserver"
Oct 29 22:42:23.981207 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.981181    4742 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:42:23.981684 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.981662    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-d55dr","kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-dqkrw","openshift-monitoring/node-exporter-tgl57","kube-system/global-pull-secret-syncer-6vdmx","kube-system/konnectivity-agent-nfh24","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx","openshift-cluster-node-tuning-operator/tuned-xj7f4","openshift-insights/insights-runtime-extractor-5vrv4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal","openshift-ingress-canary/ingress-canary-qxfxw","openshift-multus/network-metrics-daemon-f47cd","openshift-dns/dns-default-hzwmx","openshift-dns/node-resolver-xrt59","openshift-multus/multus-9rm5r","openshift-network-operator/iptables-alerter-59ghk","openshift-multus/multus-additional-cni-plugins-qh8k5","openshift-network-diagnostics/network-check-target-j26t7"]
Oct 29 22:42:23.997750 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.997727    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:42:23.998029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.998016    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:42:23.998126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.998039    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:42:23.998126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.998071    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:42:23.998126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.998071    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7wdgt\""
Oct 29 22:42:23.998126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.998104    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pdbx4\""
Oct 29 22:42:23.998261 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.998144    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:42:23.999502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.999486    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:42:23.999670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:23.999656    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:42:24.012234 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.012215    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:42:24.012770 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.012754    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kfh6g\""
Oct 29 22:42:24.012978 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.012963    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:42:24.013034 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013001    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6bbfq\""
Oct 29 22:42:24.013262 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013243    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmngh\""
Oct 29 22:42:24.013322 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013290    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7jzk\""
Oct 29 22:42:24.013470 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013449    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:42:24.013470 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013457    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g5fdr\""
Oct 29 22:42:24.013597 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013484    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4f7f9\""
Oct 29 22:42:24.013597 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013485    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:42:24.013775 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013763    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56msd\""
Oct 29 22:42:24.013828 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013802    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qj89s\""
Oct 29 22:42:24.013828 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.013812    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:42:24.014050 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.014036    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vstz\""
Oct 29 22:42:24.014050 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.014046    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bd868\""
Oct 29 22:42:24.014148 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.014050    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:42:24.014219 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.014205    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p7zhx\""
Oct 29 22:42:24.014268 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.014219    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-528cc\""
Oct 29 22:42:24.014268 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.014239    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:42:24.014334 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.014269    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pxqn\""
Oct 29 22:42:24.014448 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.014437    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gqfpm\""
Oct 29 22:42:24.015044 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.015032    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.016120 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016104    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.016165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016105    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:42:24.016165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016131    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.016165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016137    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.016614 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016584    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.016614 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016603    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.016729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016641    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:42:24.016729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016649    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.016729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016642    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:42:24.016729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016676    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.016859 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016845    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.016902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016862    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:42:24.016902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016899    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:42:24.016971 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.016902    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.023563 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.023549    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.024502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.024489    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:42:24.024587 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.024576    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.025043 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025025    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:42:24.025134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025045    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.025134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025068    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:42:24.025134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025076    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.025134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025080    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.025134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025068    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.025134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025125    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:42:24.025134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025134    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:42:24.025403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025125    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.025403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025154    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:42:24.025403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025220    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.025403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025271    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:42:24.025403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025288    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:42:24.025403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.025400    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:42:24.034889 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.034874    4742 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:24.034997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.034986    4742 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:24.054244 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.054224    4742 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:42:24.058401 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.058388    4742 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:24.058460 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:24.058424    4742 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-130.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:24.061633 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.061619    4742 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:24.061717 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:24.061657    4742 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-130.ec2.internal"
Oct 29 22:42:24.103924 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.103901    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-kubelet-config\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:24.104006 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.103929    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e048e3a-9761-4519-af80-d7e4fe16ded6-cert\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:24.104006 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.103946    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-var-lib-kubelet\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.104006 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.103963    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.104006 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.103979    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxqw\" (UniqueName: \"kubernetes.io/projected/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-api-access-tlxqw\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.104006 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.103994    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4spv\" (UniqueName: \"kubernetes.io/projected/1c02209b-22cf-4ed8-aa61-429052c91a19-kube-api-access-w4spv\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.104247 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104147    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk92n\" (UniqueName: \"kubernetes.io/projected/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-kube-api-access-sk92n\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:42:24.104292 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104282    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-config-volume\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:24.104341 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104309    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-ovn\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.104393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104333    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-log-socket\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.104393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104366    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-netd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.104393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104364    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e048e3a-9761-4519-af80-d7e4fe16ded6-cert\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:24.104517 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104445    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.104517 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104483    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.104517 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104489    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cnibin\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.104648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104519    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b117295a-3db5-4ec0-8858-43f561c7a075-serviceca\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:42:24.104648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104546    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-config\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.104648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104577    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-dbus\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:24.104648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104591    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.104648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104586    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-config-volume\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:24.104648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104630    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-netns\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104665    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-hostroot\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104712    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104751    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-systemd\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104794    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-tmp-dir\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104794    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-config\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104796    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b117295a-3db5-4ec0-8858-43f561c7a075-serviceca\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104822    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-socket-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104884    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg46r\" (UniqueName: \"kubernetes.io/projected/b117295a-3db5-4ec0-8858-43f561c7a075-kube-api-access-fg46r\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:42:24.104912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104891    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-tmp-dir\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104926    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-system-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.104958    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-cnibin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105007    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-daemon-config\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105040    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-tmp-dir\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105128    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105154    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-tmp-dir\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105183    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105269    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-iptables-alerter-script\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:42:24.105343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105318    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-daemon-config\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105347    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105354    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-hosts-file\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105424    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-slash\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105455    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105499    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-iptables-alerter-script\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105499    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-node-log\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105531    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105533    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729ef181-4607-453c-ae28-080401ddc444-metrics-certs\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105565    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/549afed0-003b-43bf-8154-49e89fcba816-konnectivity-ca\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105583    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105608    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-crio-socket\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.105675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105668    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-systemd-units\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105703    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-etc-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105731    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-tls\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105762    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b29bj\" (UniqueName: \"kubernetes.io/projected/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-kube-api-access-b29bj\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105767    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/549afed0-003b-43bf-8154-49e89fcba816-konnectivity-ca\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105802    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105815    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105819    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105838    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105857    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b117295a-3db5-4ec0-8858-43f561c7a075-host\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105886    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovn-node-metrics-cert\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105905    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbwb\" (UniqueName: \"kubernetes.io/projected/09138ffd-6ca5-4c22-bca1-fac58f4849c1-kube-api-access-9nbwb\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105943    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9lf4\" (UniqueName: \"kubernetes.io/projected/729ef181-4607-453c-ae28-080401ddc444-kube-api-access-x9lf4\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105946    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-tls\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.105992    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-socket-dir-parent\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106128    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-k8s-cni-cncf-io\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106133    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106139    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-original-pull-secret\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106156    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-kubelet\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106164    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovn-node-metrics-cert\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106197    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-multus-certs\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106224    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-netns\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.106260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106254    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-bin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106281    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tmc\" (UniqueName: \"kubernetes.io/projected/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-kube-api-access-z5tmc\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106305    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-modprobe-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106324    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysconfig\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106363    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-kubernetes\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106409    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-conf\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106445    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106475    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-bin\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106497    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctk5g\" (UniqueName: \"kubernetes.io/projected/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-kube-api-access-ctk5g\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106513    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106610    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-run\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106632    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-lib-modules\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106648    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstdq\" (UniqueName: \"kubernetes.io/projected/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-kube-api-access-gstdq\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106665    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-sys\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106684    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-root\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106703    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/549afed0-003b-43bf-8154-49e89fcba816-agent-certs\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106717    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-cni-binary-copy\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106744    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106763    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-multus\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106781    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-host-slash\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106845    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-systemd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106892    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-sys-fs\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106906    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c02209b-22cf-4ed8-aa61-429052c91a19-cni-binary-copy\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106909    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-textfile\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106967    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/549afed0-003b-43bf-8154-49e89fcba816-agent-certs\") pod \"konnectivity-agent-nfh24\" (UID: \"549afed0-003b-43bf-8154-49e89fcba816\") " pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106978    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-wtmp\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.106989    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-textfile\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107010    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-metrics-client-ca\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107039    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slqb\" (UniqueName: \"kubernetes.io/projected/4e048e3a-9761-4519-af80-d7e4fe16ded6-kube-api-access-7slqb\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:24.107061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107054    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-tmp\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107122    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-tmp\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107142    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-var-lib-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107186    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-metrics-client-ca\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107229    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74d2\" (UniqueName: \"kubernetes.io/projected/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kube-api-access-b74d2\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107256    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-system-cni-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107322    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-conf-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107371    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-etc-kubernetes\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107401    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-host\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107428    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-tuned\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107572    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-tuned\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107591    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdc7\" (UniqueName: \"kubernetes.io/projected/ba102785-5e33-4eeb-9cc8-4f03627d328a-kube-api-access-dkdc7\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107617    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-kubelet\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107646    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107670    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-device-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107714    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107756    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-env-overrides\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107800    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-metrics-tls\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107846    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-registration-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107880    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107906    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-data-volume\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107919    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107948    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-sys\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107979    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-data-volume\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107978    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.108004    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-metrics-tls\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.108008    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-script-lib\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.107977    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-env-overrides\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.108192    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.108287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.108268    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-ovnkube-script-lib\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.123977 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.123946    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk92n\" (UniqueName: \"kubernetes.io/projected/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-kube-api-access-sk92n\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:42:24.124054 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.123946    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg46r\" (UniqueName: \"kubernetes.io/projected/b117295a-3db5-4ec0-8858-43f561c7a075-kube-api-access-fg46r\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:42:24.124054 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.124000    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxqw\" (UniqueName: \"kubernetes.io/projected/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-kube-api-access-tlxqw\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.124265 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.124249    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4spv\" (UniqueName: \"kubernetes.io/projected/1c02209b-22cf-4ed8-aa61-429052c91a19-kube-api-access-w4spv\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.127543 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.127526    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b29bj\" (UniqueName: \"kubernetes.io/projected/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-kube-api-access-b29bj\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.127582 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.127533    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74d2\" (UniqueName: \"kubernetes.io/projected/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kube-api-access-b74d2\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.129482 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.129466    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbwb\" (UniqueName: \"kubernetes.io/projected/09138ffd-6ca5-4c22-bca1-fac58f4849c1-kube-api-access-9nbwb\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.130120 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.130080    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdc7\" (UniqueName: \"kubernetes.io/projected/ba102785-5e33-4eeb-9cc8-4f03627d328a-kube-api-access-dkdc7\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.130376 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.130361    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9lf4\" (UniqueName: \"kubernetes.io/projected/729ef181-4607-453c-ae28-080401ddc444-kube-api-access-x9lf4\") pod \"network-metrics-daemon-f47cd\" (UID: \"729ef181-4607-453c-ae28-080401ddc444\") " pod="openshift-multus/network-metrics-daemon-f47cd"
Oct 29 22:42:24.130550 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.130530    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstdq\" (UniqueName: \"kubernetes.io/projected/6ea35e55-0ac2-48aa-8e7d-217aae2f5445-kube-api-access-gstdq\") pod \"dns-default-hzwmx\" (UID: \"6ea35e55-0ac2-48aa-8e7d-217aae2f5445\") " pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:24.130693 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.130662    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tmc\" (UniqueName: \"kubernetes.io/projected/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-kube-api-access-z5tmc\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:42:24.130726 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.130697    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slqb\" (UniqueName: \"kubernetes.io/projected/4e048e3a-9761-4519-af80-d7e4fe16ded6-kube-api-access-7slqb\") pod \"ingress-canary-qxfxw\" (UID: \"4e048e3a-9761-4519-af80-d7e4fe16ded6\") " pod="openshift-ingress-canary/ingress-canary-qxfxw"
Oct 29 22:42:24.130826 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.130811    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvms\" (UniqueName: \"kubernetes.io/projected/7b8511e6-8125-4bdc-8542-aa58c941f70d-kube-api-access-fvvms\") pod \"network-check-target-j26t7\" (UID: \"7b8511e6-8125-4bdc-8542-aa58c941f70d\") " pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:24.131682 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.131667    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctk5g\" (UniqueName: \"kubernetes.io/projected/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-kube-api-access-ctk5g\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.208879 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208797    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-netns\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.208879 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208848    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-bin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208894    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-modprobe-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208908    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-netns\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208922    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysconfig\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208949    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-kubernetes\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208966    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-bin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208971    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysconfig\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.208976    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-conf\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209006    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-kubernetes\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209019    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-bin\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209033    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-modprobe-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209038    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-run\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209064    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-bin\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209068    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-lib-modules\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209109    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-conf\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209116    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-run\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209107    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-sys\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209141    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-root\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209160    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-root\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209141    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-sys\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209158    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-multus\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209188    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-cni-multus\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209194    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-host-slash\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209213    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-systemd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209223    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-lib-modules\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209230    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-sys-fs\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209259    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-sys-fs\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209278    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bce0d5c-59af-4502-8354-4d7fc84c4dd5-host-slash\") pod \"iptables-alerter-59ghk\" (UID: \"9bce0d5c-59af-4502-8354-4d7fc84c4dd5\") " pod="openshift-network-operator/iptables-alerter-59ghk"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209278    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-wtmp\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209289    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-systemd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209312    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-var-lib-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209338    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-system-cni-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209349    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-var-lib-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209361    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b17886d-b9f1-44b7-bcab-5fdebb8c3749-node-exporter-wtmp\") pod \"node-exporter-tgl57\" (UID: \"7b17886d-b9f1-44b7-bcab-5fdebb8c3749\") " pod="openshift-monitoring/node-exporter-tgl57"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209378    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-system-cni-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209376    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-conf-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209403    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-etc-kubernetes\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.209432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209420    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-conf-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209456    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-host\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209469    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-etc-kubernetes\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209501    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-kubelet\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209521    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209558    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209582    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-device-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209595    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-host\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209603    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-registration-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209609    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-kubelet\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209621    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-sys\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209646    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-device-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209660    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209668    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-registration-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209664    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-sys\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209704    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-kubelet-config\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209754    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-var-lib-kubelet\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209756    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209789    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-var-lib-kubelet\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209806    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-kubelet-config\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209828    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-ovn\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209844    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-log-socket\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209858    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-netd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209871    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209885    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cnibin\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209908    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-log-socket\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209909    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-dbus\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209925    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-ovn\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209950    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209954    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209959    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56ee256a-8f55-4c1d-bc82-f3cb5c3b2383-dbus\") pod \"global-pull-secret-syncer-6vdmx\" (UID: \"56ee256a-8f55-4c1d-bc82-f3cb5c3b2383\") " pod="kube-system/global-pull-secret-syncer-6vdmx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209912    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-cni-netd\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209964    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-cnibin\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.209976    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-netns\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210006    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-netns\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210010    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-hostroot\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210015    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210045    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210049    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-hostroot\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210068    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-systemd\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210132    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-socket-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210141    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-sysctl-d\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210150    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-system-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210166    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-cnibin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210177    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1e053f2-299d-4a89-be9a-bfaeb37e46f7-socket-dir\") pod \"aws-ebs-csi-driver-node-j8llx\" (UID: \"e1e053f2-299d-4a89-be9a-bfaeb37e46f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j8llx"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210142    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba102785-5e33-4eeb-9cc8-4f03627d328a-etc-systemd\") pod \"tuned-xj7f4\" (UID: \"ba102785-5e33-4eeb-9cc8-4f03627d328a\") " pod="openshift-cluster-node-tuning-operator/tuned-xj7f4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210186    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-system-cni-dir\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210190    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-hosts-file\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210210    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-cnibin\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210215    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bffb5f7d-dcee-49b1-8efb-d0314fe27f5b-hosts-file\") pod \"node-resolver-xrt59\" (UID: \"bffb5f7d-dcee-49b1-8efb-d0314fe27f5b\") " pod="openshift-dns/node-resolver-xrt59"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210222    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-slash\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210245    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210267    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-node-log\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210292    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210307    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-run-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210318    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-crio-socket\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210325    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-slash\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210320    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-node-log\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210335    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210350    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-systemd-units\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210369    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-etc-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210390    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210394    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-systemd-units\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210409    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/520fb8b2-51a7-4ecc-9a6a-21b5c608be37-crio-socket\") pod \"insights-runtime-extractor-5vrv4\" (UID: \"520fb8b2-51a7-4ecc-9a6a-21b5c608be37\") " pod="openshift-insights/insights-runtime-extractor-5vrv4"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210414    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b117295a-3db5-4ec0-8858-43f561c7a075-host\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210430    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7bb5a69-719b-48bf-a9e3-7ed728d315fa-etc-openvswitch\") pod \"ovnkube-node-dqkrw\" (UID: \"f7bb5a69-719b-48bf-a9e3-7ed728d315fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210435    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09138ffd-6ca5-4c22-bca1-fac58f4849c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qh8k5\" (UID: \"09138ffd-6ca5-4c22-bca1-fac58f4849c1\") " pod="openshift-multus/multus-additional-cni-plugins-qh8k5"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210437    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-socket-dir-parent\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210459    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b117295a-3db5-4ec0-8858-43f561c7a075-host\") pod \"node-ca-d55dr\" (UID: \"b117295a-3db5-4ec0-8858-43f561c7a075\") " pod="openshift-image-registry/node-ca-d55dr"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210480    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-multus-socket-dir-parent\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.210463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210505    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-k8s-cni-cncf-io\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.212357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210541    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-kubelet\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.212357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210566    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-multus-certs\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.212357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210591    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-multus-certs\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.212357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210611    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-run-k8s-cni-cncf-io\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:24.212357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:24.210631    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c02209b-22cf-4ed8-aa61-429052c91a19-host-var-lib-kubelet\") pod \"multus-9rm5r\" (UID: \"1c02209b-22cf-4ed8-aa61-429052c91a19\") " pod="openshift-multus/multus-9rm5r"
Oct 29 22:42:25.226824 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:25.226795    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:25.228359 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:25.228342    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hzwmx"
Oct 29 22:42:31.697457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:31.697423    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:34.032752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:34.032722    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:34.217301 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:34.217274    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:34.217492 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:34.217479    4742 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:43.139125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:43.139096    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqkrw"
Oct 29 22:42:44.245349 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:44.245325    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nfh24"
Oct 29 22:42:51.700695 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:42:51.700668    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j26t7"
Oct 29 22:42:53.121074 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:42:53.121045    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:43:23.121467 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:43:23.121429    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:43:53.120735 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:43:53.120700    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:44:23.120503 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:44:23.120468    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:44:25.172378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.172346    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"]
Oct 29 22:44:25.174249 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.174148    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"
Oct 29 22:44:25.190890 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.190865    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 29 22:44:25.190890 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.190887    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-497jd\""
Oct 29 22:44:25.191029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.190931    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 29 22:44:25.203062 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.203042    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"]
Oct 29 22:44:25.240511 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.240484    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa3196f1-3daf-41dd-8769-f12771428a11-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9995z\" (UID: \"fa3196f1-3daf-41dd-8769-f12771428a11\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"
Oct 29 22:44:25.240613 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.240515    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkgm\" (UniqueName: \"kubernetes.io/projected/fa3196f1-3daf-41dd-8769-f12771428a11-kube-api-access-7kkgm\") pod \"cert-manager-cainjector-5c57bbd4fc-9995z\" (UID: \"fa3196f1-3daf-41dd-8769-f12771428a11\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"
Oct 29 22:44:25.341327 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.341290    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa3196f1-3daf-41dd-8769-f12771428a11-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9995z\" (UID: \"fa3196f1-3daf-41dd-8769-f12771428a11\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"
Oct 29 22:44:25.341327 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.341328    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkgm\" (UniqueName: \"kubernetes.io/projected/fa3196f1-3daf-41dd-8769-f12771428a11-kube-api-access-7kkgm\") pod \"cert-manager-cainjector-5c57bbd4fc-9995z\" (UID: \"fa3196f1-3daf-41dd-8769-f12771428a11\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"
Oct 29 22:44:25.359139 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.359111    4742 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:44:25.361986 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.361966    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa3196f1-3daf-41dd-8769-f12771428a11-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9995z\" (UID: \"fa3196f1-3daf-41dd-8769-f12771428a11\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"
Oct 29 22:44:25.362108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.362077    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkgm\" (UniqueName: \"kubernetes.io/projected/fa3196f1-3daf-41dd-8769-f12771428a11-kube-api-access-7kkgm\") pod \"cert-manager-cainjector-5c57bbd4fc-9995z\" (UID: \"fa3196f1-3daf-41dd-8769-f12771428a11\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"
Oct 29 22:44:25.496494 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.496435    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"
Oct 29 22:44:25.786493 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.786418    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z"]
Oct 29 22:44:25.790379 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:44:25.790351    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa3196f1_3daf_41dd_8769_f12771428a11.slice/crio-eac2c2da8d7c54e20991228b3320b320db20cdc558f24c96742453c9c3e3aca5 WatchSource:0}: Error finding container eac2c2da8d7c54e20991228b3320b320db20cdc558f24c96742453c9c3e3aca5: Status 404 returned error can't find the container with id eac2c2da8d7c54e20991228b3320b320db20cdc558f24c96742453c9c3e3aca5
Oct 29 22:44:25.792485 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:25.792471    4742 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:44:26.391652 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:26.391621    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z" event={"ID":"fa3196f1-3daf-41dd-8769-f12771428a11","Type":"ContainerStarted","Data":"eac2c2da8d7c54e20991228b3320b320db20cdc558f24c96742453c9c3e3aca5"}
Oct 29 22:44:29.403540 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:29.403508    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z" event={"ID":"fa3196f1-3daf-41dd-8769-f12771428a11","Type":"ContainerStarted","Data":"aeaad51e8137f675717579fc9648787b3eb21e0b0bb9e598cb357a3b5540fff7"}
Oct 29 22:44:29.441439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:29.441379    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9995z" podStartSLOduration=1.303910397 podStartE2EDuration="4.441365507s" totalImagesPullingTime="3.13745511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:29.440919476 +0000 UTC m=+126.581661073" watchObservedRunningTime="2026-10-29 22:44:29.441365507 +0000 UTC m=+126.582107087"
Oct 29 22:44:33.096921 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.096887    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-bkxkd"]
Oct 29 22:44:33.098812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.098788    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bkxkd"
Oct 29 22:44:33.124734 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.124714    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lvmtd\""
Oct 29 22:44:33.125232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.125212    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bkxkd"]
Oct 29 22:44:33.187906 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.187875    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03424f48-1e0c-484e-82a2-6d7b1c410bf3-bound-sa-token\") pod \"cert-manager-597b95cb84-bkxkd\" (UID: \"03424f48-1e0c-484e-82a2-6d7b1c410bf3\") " pod="cert-manager/cert-manager-597b95cb84-bkxkd"
Oct 29 22:44:33.188040 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.187935    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjm6\" (UniqueName: \"kubernetes.io/projected/03424f48-1e0c-484e-82a2-6d7b1c410bf3-kube-api-access-4xjm6\") pod \"cert-manager-597b95cb84-bkxkd\" (UID: \"03424f48-1e0c-484e-82a2-6d7b1c410bf3\") " pod="cert-manager/cert-manager-597b95cb84-bkxkd"
Oct 29 22:44:33.288256 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.288228    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjm6\" (UniqueName: \"kubernetes.io/projected/03424f48-1e0c-484e-82a2-6d7b1c410bf3-kube-api-access-4xjm6\") pod \"cert-manager-597b95cb84-bkxkd\" (UID: \"03424f48-1e0c-484e-82a2-6d7b1c410bf3\") " pod="cert-manager/cert-manager-597b95cb84-bkxkd"
Oct 29 22:44:33.288405 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.288270    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03424f48-1e0c-484e-82a2-6d7b1c410bf3-bound-sa-token\") pod \"cert-manager-597b95cb84-bkxkd\" (UID: \"03424f48-1e0c-484e-82a2-6d7b1c410bf3\") " pod="cert-manager/cert-manager-597b95cb84-bkxkd"
Oct 29 22:44:33.307079 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.307043    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03424f48-1e0c-484e-82a2-6d7b1c410bf3-bound-sa-token\") pod \"cert-manager-597b95cb84-bkxkd\" (UID: \"03424f48-1e0c-484e-82a2-6d7b1c410bf3\") " pod="cert-manager/cert-manager-597b95cb84-bkxkd"
Oct 29 22:44:33.307200 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.307171    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjm6\" (UniqueName: \"kubernetes.io/projected/03424f48-1e0c-484e-82a2-6d7b1c410bf3-kube-api-access-4xjm6\") pod \"cert-manager-597b95cb84-bkxkd\" (UID: \"03424f48-1e0c-484e-82a2-6d7b1c410bf3\") " pod="cert-manager/cert-manager-597b95cb84-bkxkd"
Oct 29 22:44:33.408145 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.408062    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bkxkd"
Oct 29 22:44:33.696564 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:33.696479    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bkxkd"]
Oct 29 22:44:33.699750 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:44:33.699723    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03424f48_1e0c_484e_82a2_6d7b1c410bf3.slice/crio-87f87ed6c3806db17b7cfce9c7f31fbf4c297a85b29e0d82cae8923a4a4a4630 WatchSource:0}: Error finding container 87f87ed6c3806db17b7cfce9c7f31fbf4c297a85b29e0d82cae8923a4a4a4630: Status 404 returned error can't find the container with id 87f87ed6c3806db17b7cfce9c7f31fbf4c297a85b29e0d82cae8923a4a4a4630
Oct 29 22:44:34.419699 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:34.419665    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bkxkd" event={"ID":"03424f48-1e0c-484e-82a2-6d7b1c410bf3","Type":"ContainerStarted","Data":"7982f5662183d7f871607496efddb4f08d5021525927d835ec08e6b323df1513"}
Oct 29 22:44:34.420108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:34.419707    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bkxkd" event={"ID":"03424f48-1e0c-484e-82a2-6d7b1c410bf3","Type":"ContainerStarted","Data":"87f87ed6c3806db17b7cfce9c7f31fbf4c297a85b29e0d82cae8923a4a4a4630"}
Oct 29 22:44:34.458895 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:34.458850    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-bkxkd" podStartSLOduration=1.458837028 podStartE2EDuration="1.458837028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:34.458715094 +0000 UTC m=+131.599456677" watchObservedRunningTime="2026-10-29 22:44:34.458837028 +0000 UTC m=+131.599578608"
Oct 29 22:44:53.120951 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:44:53.120877    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:44:57.773415 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.773317    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"]
Oct 29 22:44:57.775973 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.775955    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.795218 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.792289    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 22:44:57.803439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.803414    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 29 22:44:57.803554 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.803485    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 22:44:57.803554 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.803495    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 22:44:57.816439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.816415    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"]
Oct 29 22:44:57.837353 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.837317    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrgf\" (UniqueName: \"kubernetes.io/projected/6d357a54-17bf-4d65-85db-134022425801-kube-api-access-2jrgf\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.837475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.837359    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d357a54-17bf-4d65-85db-134022425801-tmp\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.837475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.837443    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-metrics-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.837610 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.837478    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d357a54-17bf-4d65-85db-134022425801-manifests\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.837610 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.837512    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.837671 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.837622    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-webhook-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.895935 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.895901    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 22:44:57.896144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.896118    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k79nx\""
Oct 29 22:44:57.938843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.938809    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrgf\" (UniqueName: \"kubernetes.io/projected/6d357a54-17bf-4d65-85db-134022425801-kube-api-access-2jrgf\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.938843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.938845    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d357a54-17bf-4d65-85db-134022425801-tmp\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.939038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.938869    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-metrics-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.939038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.938893    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d357a54-17bf-4d65-85db-134022425801-manifests\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.939038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.938932    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.939038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.938962    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-webhook-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.939388 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.939367    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d357a54-17bf-4d65-85db-134022425801-manifests\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.941317 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.941283    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d357a54-17bf-4d65-85db-134022425801-tmp\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.941408 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.941288    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-webhook-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.941408 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.941388    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-metrics-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.941490 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.941408    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d357a54-17bf-4d65-85db-134022425801-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:57.957164 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:57.957140    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrgf\" (UniqueName: \"kubernetes.io/projected/6d357a54-17bf-4d65-85db-134022425801-kube-api-access-2jrgf\") pod \"opendatahub-operator-controller-manager-675b698459-nn6hx\" (UID: \"6d357a54-17bf-4d65-85db-134022425801\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:58.088933 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:58.088873    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:44:58.442839 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:58.442808    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"]
Oct 29 22:44:58.445146 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:44:58.445120    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d357a54_17bf_4d65_85db_134022425801.slice/crio-f2f39c06ed118791f00e239b9766bddf53dbd9e2864d5ee98fd36d99ec8a3978 WatchSource:0}: Error finding container f2f39c06ed118791f00e239b9766bddf53dbd9e2864d5ee98fd36d99ec8a3978: Status 404 returned error can't find the container with id f2f39c06ed118791f00e239b9766bddf53dbd9e2864d5ee98fd36d99ec8a3978
Oct 29 22:44:58.488592 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:44:58.488559    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx" event={"ID":"6d357a54-17bf-4d65-85db-134022425801","Type":"ContainerStarted","Data":"f2f39c06ed118791f00e239b9766bddf53dbd9e2864d5ee98fd36d99ec8a3978"}
Oct 29 22:45:01.500168 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:01.500074    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx" event={"ID":"6d357a54-17bf-4d65-85db-134022425801","Type":"ContainerStarted","Data":"347039e46d81ec9f6cae9a42c00086bfaf3d0b278175789c605eecdf9a6df59d"}
Oct 29 22:45:02.504153 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:02.504119    4742 generic.go:356] "Generic (PLEG): container finished" podID="6d357a54-17bf-4d65-85db-134022425801" containerID="347039e46d81ec9f6cae9a42c00086bfaf3d0b278175789c605eecdf9a6df59d" exitCode=0
Oct 29 22:45:02.504538 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:02.504177    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx" event={"ID":"6d357a54-17bf-4d65-85db-134022425801","Type":"ContainerDied","Data":"347039e46d81ec9f6cae9a42c00086bfaf3d0b278175789c605eecdf9a6df59d"}
Oct 29 22:45:03.509009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:03.508969    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx" event={"ID":"6d357a54-17bf-4d65-85db-134022425801","Type":"ContainerStarted","Data":"7835ac59d245db0e5a0172e2cd2827b552cf05794d234ad5d4ccda54f9fd8232"}
Oct 29 22:45:03.509518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:03.509123    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:45:03.664103 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:03.664046    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx" podStartSLOduration=2.966530835 podStartE2EDuration="6.664031489s" totalImagesPullingTime="3.232668885s" totalInitContainerRuntime="464.831769ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.651661897 +0000 UTC m=+160.792403478" watchObservedRunningTime="2026-10-29 22:45:03.664031489 +0000 UTC m=+160.804773071"
Oct 29 22:45:14.515127 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:14.515099    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-nn6hx"
Oct 29 22:45:17.693470 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.693440    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"]
Oct 29 22:45:17.698312 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.698295    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.725772 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.725747    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"]
Oct 29 22:45:17.750843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.750816    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6fhn4\""
Oct 29 22:45:17.750843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.750816    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 22:45:17.750843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.750820    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 22:45:17.793277 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.793245    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d625bfef-ab90-429b-844f-d4551e8d1c8d-tls-certs\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.793437 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.793305    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d625bfef-ab90-429b-844f-d4551e8d1c8d-tmp\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.793437 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.793377    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflft\" (UniqueName: \"kubernetes.io/projected/d625bfef-ab90-429b-844f-d4551e8d1c8d-kube-api-access-kflft\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.802743 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.802724    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 22:45:17.803060 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.803047    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 22:45:17.894459 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.894423    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d625bfef-ab90-429b-844f-d4551e8d1c8d-tmp\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.894553 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.894465    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflft\" (UniqueName: \"kubernetes.io/projected/d625bfef-ab90-429b-844f-d4551e8d1c8d-kube-api-access-kflft\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.894611 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.894565    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d625bfef-ab90-429b-844f-d4551e8d1c8d-tls-certs\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.896720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.896703    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d625bfef-ab90-429b-844f-d4551e8d1c8d-tmp\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.896798 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.896783    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d625bfef-ab90-429b-844f-d4551e8d1c8d-tls-certs\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:17.914798 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:17.914771    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflft\" (UniqueName: \"kubernetes.io/projected/d625bfef-ab90-429b-844f-d4551e8d1c8d-kube-api-access-kflft\") pod \"kube-auth-proxy-84d6d655bb-2n6wv\" (UID: \"d625bfef-ab90-429b-844f-d4551e8d1c8d\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:18.009969 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.009898    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"
Oct 29 22:45:18.300595 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.300524    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv"]
Oct 29 22:45:18.303730 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:45:18.303699    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd625bfef_ab90_429b_844f_d4551e8d1c8d.slice/crio-fa08b94a85a4414bb5874679f50cd9e43c782e912e5e60582337792242bb382a WatchSource:0}: Error finding container fa08b94a85a4414bb5874679f50cd9e43c782e912e5e60582337792242bb382a: Status 404 returned error can't find the container with id fa08b94a85a4414bb5874679f50cd9e43c782e912e5e60582337792242bb382a
Oct 29 22:45:18.553400 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.553319    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv" event={"ID":"d625bfef-ab90-429b-844f-d4551e8d1c8d","Type":"ContainerStarted","Data":"fa08b94a85a4414bb5874679f50cd9e43c782e912e5e60582337792242bb382a"}
Oct 29 22:45:18.883052 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.883025    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"]
Oct 29 22:45:18.887260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.887244    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:18.914483 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.914448    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"]
Oct 29 22:45:18.987154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.987131    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 29 22:45:18.987335 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.987304    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:45:18.987389 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:18.987352    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 29 22:45:19.001689 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.001659    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aad1f7d9-40f9-4a53-b8e3-7d655648543f-manager-config\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.001796 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.001693    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aad1f7d9-40f9-4a53-b8e3-7d655648543f-metrics-cert\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.001796 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.001715    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rh6v\" (UniqueName: \"kubernetes.io/projected/aad1f7d9-40f9-4a53-b8e3-7d655648543f-kube-api-access-2rh6v\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.001890 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.001838    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad1f7d9-40f9-4a53-b8e3-7d655648543f-cert\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.015704 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.015668    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 29 22:45:19.015704 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.015668    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 29 22:45:19.015861 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.015822    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cnhp8\""
Oct 29 22:45:19.102369 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.102327    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rh6v\" (UniqueName: \"kubernetes.io/projected/aad1f7d9-40f9-4a53-b8e3-7d655648543f-kube-api-access-2rh6v\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.102555 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.102404    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad1f7d9-40f9-4a53-b8e3-7d655648543f-cert\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.102555 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.102496    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aad1f7d9-40f9-4a53-b8e3-7d655648543f-manager-config\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.102555 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.102521    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aad1f7d9-40f9-4a53-b8e3-7d655648543f-metrics-cert\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.104002 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.103979    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aad1f7d9-40f9-4a53-b8e3-7d655648543f-manager-config\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.105614 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.105586    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aad1f7d9-40f9-4a53-b8e3-7d655648543f-metrics-cert\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.111602 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.111576    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad1f7d9-40f9-4a53-b8e3-7d655648543f-cert\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.123722 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.123692    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rh6v\" (UniqueName: \"kubernetes.io/projected/aad1f7d9-40f9-4a53-b8e3-7d655648543f-kube-api-access-2rh6v\") pod \"lws-controller-manager-59b8cf9b84-5nmlt\" (UID: \"aad1f7d9-40f9-4a53-b8e3-7d655648543f\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.198291 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.198201    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:19.518462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.518379    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"]
Oct 29 22:45:19.523350 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:45:19.523319    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad1f7d9_40f9_4a53_b8e3_7d655648543f.slice/crio-a2705dc648bd3ffa2474a96d9191cd79446f02de2ae938f75681e2dc284009e7 WatchSource:0}: Error finding container a2705dc648bd3ffa2474a96d9191cd79446f02de2ae938f75681e2dc284009e7: Status 404 returned error can't find the container with id a2705dc648bd3ffa2474a96d9191cd79446f02de2ae938f75681e2dc284009e7
Oct 29 22:45:19.559039 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:19.558996    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt" event={"ID":"aad1f7d9-40f9-4a53-b8e3-7d655648543f","Type":"ContainerStarted","Data":"a2705dc648bd3ffa2474a96d9191cd79446f02de2ae938f75681e2dc284009e7"}
Oct 29 22:45:21.569560 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:21.569527    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv" event={"ID":"d625bfef-ab90-429b-844f-d4551e8d1c8d","Type":"ContainerStarted","Data":"c82784cbbd149e2a5efe18fb8f19f1432a47852b474f0d43fcb4570424acdd5a"}
Oct 29 22:45:21.609400 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:21.609347    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-2n6wv" podStartSLOduration=2.047222181 podStartE2EDuration="4.609330723s" totalImagesPullingTime="2.562108542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:21.608164409 +0000 UTC m=+178.748905992" watchObservedRunningTime="2026-10-29 22:45:21.609330723 +0000 UTC m=+178.750072308"
Oct 29 22:45:22.574047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:22.573963    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt" event={"ID":"aad1f7d9-40f9-4a53-b8e3-7d655648543f","Type":"ContainerStarted","Data":"d3e2ce426ecb903638ca3bfdac9d7008615c1da43cb13be5b9b289e99f76f853"}
Oct 29 22:45:22.574449 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:22.574146    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:22.611160 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:22.611114    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt" podStartSLOduration=1.874528802 podStartE2EDuration="4.61110082s" totalImagesPullingTime="2.736572018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:22.610683992 +0000 UTC m=+179.751425572" watchObservedRunningTime="2026-10-29 22:45:22.61110082 +0000 UTC m=+179.751842395"
Oct 29 22:45:23.122978 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:45:23.122942    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:45:23.527515 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.527450    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"]
Oct 29 22:45:23.532234 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.532214    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.546989 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.546969    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8vxxz\""
Oct 29 22:45:23.547101 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.547055    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 22:45:23.561186 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.561160    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"]
Oct 29 22:45:23.636073 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636036    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.636457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636081    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c571f4f4-be21-4c94-866b-cb67dd505862-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.636457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636183    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.636457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636311    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.636457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636335    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c571f4f4-be21-4c94-866b-cb67dd505862-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.636457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636366    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.636457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636416    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c571f4f4-be21-4c94-866b-cb67dd505862-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.636712 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636468    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n5t7\" (UniqueName: \"kubernetes.io/projected/c571f4f4-be21-4c94-866b-cb67dd505862-kube-api-access-8n5t7\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.636712 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.636527    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737316 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737283    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c571f4f4-be21-4c94-866b-cb67dd505862-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737326    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n5t7\" (UniqueName: \"kubernetes.io/projected/c571f4f4-be21-4c94-866b-cb67dd505862-kube-api-access-8n5t7\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737359    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737395    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737434    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c571f4f4-be21-4c94-866b-cb67dd505862-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737626 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737460    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737626 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737494    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737626 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737518    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c571f4f4-be21-4c94-866b-cb67dd505862-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737626 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737561    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.737918 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.737895    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.738025 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.738001    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.738140 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.738118    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.738201 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.738140    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.738247 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.738220    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c571f4f4-be21-4c94-866b-cb67dd505862-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.739679 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.739659    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c571f4f4-be21-4c94-866b-cb67dd505862-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.739807 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.739790    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c571f4f4-be21-4c94-866b-cb67dd505862-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.756036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.756007    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c571f4f4-be21-4c94-866b-cb67dd505862-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.756432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.756414    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n5t7\" (UniqueName: \"kubernetes.io/projected/c571f4f4-be21-4c94-866b-cb67dd505862-kube-api-access-8n5t7\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f44nnj\" (UID: \"c571f4f4-be21-4c94-866b-cb67dd505862\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:23.844442 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:23.844404    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:24.150606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:24.150533    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"]
Oct 29 22:45:24.153830 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:45:24.153805    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc571f4f4_be21_4c94_866b_cb67dd505862.slice/crio-a2555412c3c7bf2018c3094de86a551a1230ca79ea732ec9c7d25dd247847d16 WatchSource:0}: Error finding container a2555412c3c7bf2018c3094de86a551a1230ca79ea732ec9c7d25dd247847d16: Status 404 returned error can't find the container with id a2555412c3c7bf2018c3094de86a551a1230ca79ea732ec9c7d25dd247847d16
Oct 29 22:45:24.581253 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:24.581165    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj" event={"ID":"c571f4f4-be21-4c94-866b-cb67dd505862","Type":"ContainerStarted","Data":"a2555412c3c7bf2018c3094de86a551a1230ca79ea732ec9c7d25dd247847d16"}
Oct 29 22:45:27.270780 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:27.270749    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:45:27.271055 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:27.270814    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:45:27.271055 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:27.270844    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:45:27.593036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:27.592999    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj" event={"ID":"c571f4f4-be21-4c94-866b-cb67dd505862","Type":"ContainerStarted","Data":"d2b45838a03abfdf6b870e7e9a5c5c8e78a37312da887737e92b37a1f5ce1d22"}
Oct 29 22:45:27.632636 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:27.632587    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj" podStartSLOduration=1.518342302 podStartE2EDuration="4.632571085s" totalImagesPullingTime="3.114228783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:27.632260783 +0000 UTC m=+184.773002366" watchObservedRunningTime="2026-10-29 22:45:27.632571085 +0000 UTC m=+184.773312674"
Oct 29 22:45:27.845116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:27.845018    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:28.849764 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:28.849737    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:29.599216 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:29.599180    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:29.600118 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:29.600077    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f44nnj"
Oct 29 22:45:33.578988 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.578960    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-5nmlt"
Oct 29 22:45:33.804948 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.804913    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"]
Oct 29 22:45:33.808244 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.808226    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:33.827100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.827059    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-vbbw5\""
Oct 29 22:45:33.851302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.851269    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 29 22:45:33.852186 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.852164    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"]
Oct 29 22:45:33.916851 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.916809    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:33.916990 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.916882    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwsp\" (UniqueName: \"kubernetes.io/projected/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-kube-api-access-wjwsp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:33.916990 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.916979    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:33.917115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:33.917006    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.017955 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.017920    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwsp\" (UniqueName: \"kubernetes.io/projected/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-kube-api-access-wjwsp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.018153 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.017967    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.018153 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.017987    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.018153 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.018020    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.018411 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.018390    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.018470 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.018436    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.019049 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.019029    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.037320 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.037297    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwsp\" (UniqueName: \"kubernetes.io/projected/ce0f0afa-5174-44d2-a80e-990ce4e0ec48-kube-api-access-wjwsp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ldmzq\" (UID: \"ce0f0afa-5174-44d2-a80e-990ce4e0ec48\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.120606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.120528    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:34.443115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.443015    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"]
Oct 29 22:45:34.447034 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:45:34.447004    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0f0afa_5174_44d2_a80e_990ce4e0ec48.slice/crio-b776f8d7b644735171ae11ec2f8ac11d7714e4797397d7bd885f4c52d7051427 WatchSource:0}: Error finding container b776f8d7b644735171ae11ec2f8ac11d7714e4797397d7bd885f4c52d7051427: Status 404 returned error can't find the container with id b776f8d7b644735171ae11ec2f8ac11d7714e4797397d7bd885f4c52d7051427
Oct 29 22:45:34.615926 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:34.615893    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq" event={"ID":"ce0f0afa-5174-44d2-a80e-990ce4e0ec48","Type":"ContainerStarted","Data":"b776f8d7b644735171ae11ec2f8ac11d7714e4797397d7bd885f4c52d7051427"}
Oct 29 22:45:36.623741 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:36.623709    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq" event={"ID":"ce0f0afa-5174-44d2-a80e-990ce4e0ec48","Type":"ContainerStarted","Data":"2201b3e399425b8ef8e364fbffe75d187486cd3a1e0e479e369c82572fd217be"}
Oct 29 22:45:36.624154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:36.623825    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:53.123625 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:45:53.123588    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:45:57.668635 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:57.668607    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq"
Oct 29 22:45:57.723789 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:45:57.720768    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ldmzq" podStartSLOduration=23.293363776 podStartE2EDuration="24.720746015s" totalImagesPullingTime="1.427382239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:36.686881809 +0000 UTC m=+193.827623394" watchObservedRunningTime="2026-10-29 22:45:57.720746015 +0000 UTC m=+214.861487600"
Oct 29 22:46:20.234812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.234772    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-65h2w"]
Oct 29 22:46:20.239815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.239790    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:20.247563 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.247542    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d61643b2-b8ab-4251-95db-73e3969f05dd-cert\") pod \"kserve-controller-manager-776cc66d47-65h2w\" (UID: \"d61643b2-b8ab-4251-95db-73e3969f05dd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:20.247638 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.247582    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6db2q\" (UniqueName: \"kubernetes.io/projected/d61643b2-b8ab-4251-95db-73e3969f05dd-kube-api-access-6db2q\") pod \"kserve-controller-manager-776cc66d47-65h2w\" (UID: \"d61643b2-b8ab-4251-95db-73e3969f05dd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:20.269496 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.269471    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 29 22:46:20.269780 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.269767    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vbb2z\""
Oct 29 22:46:20.270734 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.270706    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-65h2w"]
Oct 29 22:46:20.348342 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.348305    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6db2q\" (UniqueName: \"kubernetes.io/projected/d61643b2-b8ab-4251-95db-73e3969f05dd-kube-api-access-6db2q\") pod \"kserve-controller-manager-776cc66d47-65h2w\" (UID: \"d61643b2-b8ab-4251-95db-73e3969f05dd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:20.348512 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.348377    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d61643b2-b8ab-4251-95db-73e3969f05dd-cert\") pod \"kserve-controller-manager-776cc66d47-65h2w\" (UID: \"d61643b2-b8ab-4251-95db-73e3969f05dd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:20.350854 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.350822    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d61643b2-b8ab-4251-95db-73e3969f05dd-cert\") pod \"kserve-controller-manager-776cc66d47-65h2w\" (UID: \"d61643b2-b8ab-4251-95db-73e3969f05dd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:20.374828 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.374800    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6db2q\" (UniqueName: \"kubernetes.io/projected/d61643b2-b8ab-4251-95db-73e3969f05dd-kube-api-access-6db2q\") pod \"kserve-controller-manager-776cc66d47-65h2w\" (UID: \"d61643b2-b8ab-4251-95db-73e3969f05dd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:20.380886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.380853    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-nhwrp"]
Oct 29 22:46:20.382972 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.382952    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.418830 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.418793    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-nhwrp"]
Oct 29 22:46:20.449008 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.448973    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-metrics-certs\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.449197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.449022    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-cert\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.449197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.449079    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fc9j\" (UniqueName: \"kubernetes.io/projected/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-kube-api-access-9fc9j\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.463520 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.463491    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5f6sg\""
Oct 29 22:46:20.463664 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.463573    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 29 22:46:20.463664 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.463574    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 29 22:46:20.550187 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.550119    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-cert\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.550187 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.550156    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fc9j\" (UniqueName: \"kubernetes.io/projected/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-kube-api-access-9fc9j\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.550371 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.550241    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:20.550371 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.550294    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-metrics-certs\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.552699 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.552670    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-cert\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.552837 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.552674    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-metrics-certs\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.578432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.578409    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fc9j\" (UniqueName: \"kubernetes.io/projected/c2b0ebb8-03da-4288-a806-a100d8e6ff1c-kube-api-access-9fc9j\") pod \"odh-model-controller-f87fcf56c-nhwrp\" (UID: \"c2b0ebb8-03da-4288-a806-a100d8e6ff1c\") " pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.695703 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.695669    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:20.843055 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.842951    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-65h2w"]
Oct 29 22:46:20.845552 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:46:20.845527    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61643b2_b8ab_4251_95db_73e3969f05dd.slice/crio-ad951b87433033a4efb4471dd4313c42f928cc9588c34245e1838aed7ff79cfc WatchSource:0}: Error finding container ad951b87433033a4efb4471dd4313c42f928cc9588c34245e1838aed7ff79cfc: Status 404 returned error can't find the container with id ad951b87433033a4efb4471dd4313c42f928cc9588c34245e1838aed7ff79cfc
Oct 29 22:46:20.992746 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:20.992718    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-nhwrp"]
Oct 29 22:46:20.992872 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:46:20.992803    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b0ebb8_03da_4288_a806_a100d8e6ff1c.slice/crio-685caf18e01df0c7cb3d90752185c34a04ab87e99e9ddee4bdc824ea57d90c2f WatchSource:0}: Error finding container 685caf18e01df0c7cb3d90752185c34a04ab87e99e9ddee4bdc824ea57d90c2f: Status 404 returned error can't find the container with id 685caf18e01df0c7cb3d90752185c34a04ab87e99e9ddee4bdc824ea57d90c2f
Oct 29 22:46:21.767291 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:21.767188    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp" event={"ID":"c2b0ebb8-03da-4288-a806-a100d8e6ff1c","Type":"ContainerStarted","Data":"685caf18e01df0c7cb3d90752185c34a04ab87e99e9ddee4bdc824ea57d90c2f"}
Oct 29 22:46:21.771027 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:21.770992    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w" event={"ID":"d61643b2-b8ab-4251-95db-73e3969f05dd","Type":"ContainerStarted","Data":"ad951b87433033a4efb4471dd4313c42f928cc9588c34245e1838aed7ff79cfc"}
Oct 29 22:46:23.134407 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:46:23.133920    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:46:23.782095 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:23.782047    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp" event={"ID":"c2b0ebb8-03da-4288-a806-a100d8e6ff1c","Type":"ContainerStarted","Data":"4ce89201febf98ffd76bdd54d852bf5ab2f6b74de9d167a73b4b27ad6ab8e347"}
Oct 29 22:46:23.782270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:23.782149    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:23.783485 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:23.783463    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w" event={"ID":"d61643b2-b8ab-4251-95db-73e3969f05dd","Type":"ContainerStarted","Data":"1af0d13b5360e79894b8648f78fb30b53adafeaf3b2bd3465caf0debfc74dac0"}
Oct 29 22:46:23.783605 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:23.783589    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:46:23.828662 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:23.828583    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp" podStartSLOduration=1.279045502 podStartE2EDuration="3.828571503s" totalImagesPullingTime="2.549526001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:23.828125575 +0000 UTC m=+240.968867159" watchObservedRunningTime="2026-10-29 22:46:23.828571503 +0000 UTC m=+240.969313086"
Oct 29 22:46:23.875288 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:23.875248    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w" podStartSLOduration=1.6510310540000002 podStartE2EDuration="3.875235496s" totalImagesPullingTime="2.224204442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:23.873537803 +0000 UTC m=+241.014279388" watchObservedRunningTime="2026-10-29 22:46:23.875235496 +0000 UTC m=+241.015977080"
Oct 29 22:46:34.788588 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:34.788556    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-nhwrp"
Oct 29 22:46:53.122011 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:46:53.121973    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:46:54.791835 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:46:54.791799    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-65h2w"
Oct 29 22:47:23.121655 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:47:23.121617    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:47:53.122909 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:47:53.122825    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:48:23.122268 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:48:23.122231    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:48:37.611243 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.611212    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m7hkg"]
Oct 29 22:48:37.614306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.614281    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m7hkg"
Oct 29 22:48:37.631802 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.631773    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m7hkg"]
Oct 29 22:48:37.642221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.642198    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 22:48:37.642333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.642250    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 22:48:37.642333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.642268    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xkx8\""
Oct 29 22:48:37.683270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.683245    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2pd\" (UniqueName: \"kubernetes.io/projected/0b5044d4-bdce-4411-88f6-07c0161c5c5f-kube-api-access-vh2pd\") pod \"authorino-57f7f8659f-m7hkg\" (UID: \"0b5044d4-bdce-4411-88f6-07c0161c5c5f\") " pod="kuadrant-system/authorino-57f7f8659f-m7hkg"
Oct 29 22:48:37.784565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.784535    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2pd\" (UniqueName: \"kubernetes.io/projected/0b5044d4-bdce-4411-88f6-07c0161c5c5f-kube-api-access-vh2pd\") pod \"authorino-57f7f8659f-m7hkg\" (UID: \"0b5044d4-bdce-4411-88f6-07c0161c5c5f\") " pod="kuadrant-system/authorino-57f7f8659f-m7hkg"
Oct 29 22:48:37.804257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.804221    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2pd\" (UniqueName: \"kubernetes.io/projected/0b5044d4-bdce-4411-88f6-07c0161c5c5f-kube-api-access-vh2pd\") pod \"authorino-57f7f8659f-m7hkg\" (UID: \"0b5044d4-bdce-4411-88f6-07c0161c5c5f\") " pod="kuadrant-system/authorino-57f7f8659f-m7hkg"
Oct 29 22:48:37.925032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:37.924954    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m7hkg"
Oct 29 22:48:38.214609 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.214539    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m7hkg"]
Oct 29 22:48:38.217856 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:48:38.217818    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b5044d4_bdce_4411_88f6_07c0161c5c5f.slice/crio-5ffd9e323561d10a38d6025211eb6355e85151d86fc585e878b864f24ae405d7 WatchSource:0}: Error finding container 5ffd9e323561d10a38d6025211eb6355e85151d86fc585e878b864f24ae405d7: Status 404 returned error can't find the container with id 5ffd9e323561d10a38d6025211eb6355e85151d86fc585e878b864f24ae405d7
Oct 29 22:48:38.265366 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.265341    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qrss"]
Oct 29 22:48:38.269499 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.269481    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss"
Oct 29 22:48:38.283148 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.283129    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qrss"]
Oct 29 22:48:38.390404 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.390362    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5h5\" (UniqueName: \"kubernetes.io/projected/05c6cc3c-835b-4aff-9613-65ec7b1bddd5-kube-api-access-zs5h5\") pod \"authorino-56c6c9ddd5-4qrss\" (UID: \"05c6cc3c-835b-4aff-9613-65ec7b1bddd5\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qrss"
Oct 29 22:48:38.490883 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.490802    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5h5\" (UniqueName: \"kubernetes.io/projected/05c6cc3c-835b-4aff-9613-65ec7b1bddd5-kube-api-access-zs5h5\") pod \"authorino-56c6c9ddd5-4qrss\" (UID: \"05c6cc3c-835b-4aff-9613-65ec7b1bddd5\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qrss"
Oct 29 22:48:38.509697 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.509664    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5h5\" (UniqueName: \"kubernetes.io/projected/05c6cc3c-835b-4aff-9613-65ec7b1bddd5-kube-api-access-zs5h5\") pod \"authorino-56c6c9ddd5-4qrss\" (UID: \"05c6cc3c-835b-4aff-9613-65ec7b1bddd5\") " pod="kuadrant-system/authorino-56c6c9ddd5-4qrss"
Oct 29 22:48:38.580341 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.580313    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss"
Oct 29 22:48:38.869652 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:38.869622    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qrss"]
Oct 29 22:48:38.873959 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:48:38.873931    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05c6cc3c_835b_4aff_9613_65ec7b1bddd5.slice/crio-256e9de17b062ce6e34067a990aef5bbd8b76885b618806ec1b30a6899965c3e WatchSource:0}: Error finding container 256e9de17b062ce6e34067a990aef5bbd8b76885b618806ec1b30a6899965c3e: Status 404 returned error can't find the container with id 256e9de17b062ce6e34067a990aef5bbd8b76885b618806ec1b30a6899965c3e
Oct 29 22:48:39.221615 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:39.221484    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss" event={"ID":"05c6cc3c-835b-4aff-9613-65ec7b1bddd5","Type":"ContainerStarted","Data":"256e9de17b062ce6e34067a990aef5bbd8b76885b618806ec1b30a6899965c3e"}
Oct 29 22:48:39.222956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:39.222919    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m7hkg" event={"ID":"0b5044d4-bdce-4411-88f6-07c0161c5c5f","Type":"ContainerStarted","Data":"5ffd9e323561d10a38d6025211eb6355e85151d86fc585e878b864f24ae405d7"}
Oct 29 22:48:41.232942 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:41.232851    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss" event={"ID":"05c6cc3c-835b-4aff-9613-65ec7b1bddd5","Type":"ContainerStarted","Data":"7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3"}
Oct 29 22:48:41.234350 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:41.234322    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m7hkg" event={"ID":"0b5044d4-bdce-4411-88f6-07c0161c5c5f","Type":"ContainerStarted","Data":"0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a"}
Oct 29 22:48:41.271321 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:41.271278    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss" podStartSLOduration=1.220321909 podStartE2EDuration="3.271266252s" totalImagesPullingTime="2.050944343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:41.270830324 +0000 UTC m=+378.411571908" watchObservedRunningTime="2026-10-29 22:48:41.271266252 +0000 UTC m=+378.412007836"
Oct 29 22:48:41.305484 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:41.305454    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m7hkg"]
Oct 29 22:48:41.321437 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:41.321394    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-m7hkg" podStartSLOduration=1.6045098169999998 podStartE2EDuration="4.321382056s" totalImagesPullingTime="2.716872239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:41.320682399 +0000 UTC m=+378.461423983" watchObservedRunningTime="2026-10-29 22:48:41.321382056 +0000 UTC m=+378.462123640"
Oct 29 22:48:43.241720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:43.241681    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-m7hkg" podUID="0b5044d4-bdce-4411-88f6-07c0161c5c5f" containerName="authorino" containerID="cri-o://0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a" gracePeriod=30
Oct 29 22:48:43.472536 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:43.472512    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m7hkg"
Oct 29 22:48:43.633488 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:43.633460    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5044d4-bdce-4411-88f6-07c0161c5c5f-kube-api-access-vh2pd\" (UniqueName: \"kubernetes.io/projected/0b5044d4-bdce-4411-88f6-07c0161c5c5f-kube-api-access-vh2pd\") pod \"0b5044d4-bdce-4411-88f6-07c0161c5c5f\" (UID: \"0b5044d4-bdce-4411-88f6-07c0161c5c5f\") "
Oct 29 22:48:43.635516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:43.635487    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5044d4-bdce-4411-88f6-07c0161c5c5f-kube-api-access-vh2pd" pod "0b5044d4-bdce-4411-88f6-07c0161c5c5f" (UID: "0b5044d4-bdce-4411-88f6-07c0161c5c5f"). InnerVolumeSpecName "kube-api-access-vh2pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:43.734197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:43.734153    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2pd\" (UniqueName: \"kubernetes.io/projected/0b5044d4-bdce-4411-88f6-07c0161c5c5f-kube-api-access-vh2pd\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:48:44.246064 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.246035    4742 generic.go:356] "Generic (PLEG): container finished" podID="0b5044d4-bdce-4411-88f6-07c0161c5c5f" containerID="0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a" exitCode=0
Oct 29 22:48:44.246472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.246072    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m7hkg" event={"ID":"0b5044d4-bdce-4411-88f6-07c0161c5c5f","Type":"ContainerDied","Data":"0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a"}
Oct 29 22:48:44.246472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.246114    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m7hkg" event={"ID":"0b5044d4-bdce-4411-88f6-07c0161c5c5f","Type":"ContainerDied","Data":"5ffd9e323561d10a38d6025211eb6355e85151d86fc585e878b864f24ae405d7"}
Oct 29 22:48:44.246472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.246132    4742 scope.go:122] "RemoveContainer" containerID="0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a"
Oct 29 22:48:44.246472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.246074    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m7hkg"
Oct 29 22:48:44.254603 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.254583    4742 scope.go:122] "RemoveContainer" containerID="0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a"
Oct 29 22:48:44.254880 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:48:44.254861    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a\": container with ID starting with 0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a not found: ID does not exist" containerID="0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a"
Oct 29 22:48:44.254926 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.254889    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a"} err="failed to get container status \"0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a\": rpc error: code = NotFound desc = could not find container \"0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a\": container with ID starting with 0cca447c68cbdf1453645abb30dcf1dd4adeb70e499fd73b971a84956065018a not found: ID does not exist"
Oct 29 22:48:44.295838 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.295807    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m7hkg"]
Oct 29 22:48:44.307042 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.307015    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m7hkg"]
Oct 29 22:48:44.988860 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:48:44.988828    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5044d4-bdce-4411-88f6-07c0161c5c5f" path="/var/lib/kubelet/pods/0b5044d4-bdce-4411-88f6-07c0161c5c5f/volumes"
Oct 29 22:48:53.121414 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:48:53.121377    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:49:07.909638 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:07.909556    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cjc98"]
Oct 29 22:49:07.910009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:07.909808    4742 cpu_manager.go:414] "containerMap: removing container" podUID="0b5044d4-bdce-4411-88f6-07c0161c5c5f" containerName="authorino"
Oct 29 22:49:07.910009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:07.909819    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5044d4-bdce-4411-88f6-07c0161c5c5f" containerName="authorino"
Oct 29 22:49:07.910009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:07.909896    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5044d4-bdce-4411-88f6-07c0161c5c5f" containerName="authorino"
Oct 29 22:49:07.912593 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:07.912572    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cjc98"
Oct 29 22:49:07.929484 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:07.929461    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cjc98"]
Oct 29 22:49:08.013320 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.013290    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtg7\" (UniqueName: \"kubernetes.io/projected/beb265a7-e30c-48a1-adb9-f85d04d4604f-kube-api-access-cxtg7\") pod \"authorino-6f6944bcdd-cjc98\" (UID: \"beb265a7-e30c-48a1-adb9-f85d04d4604f\") " pod="kuadrant-system/authorino-6f6944bcdd-cjc98"
Oct 29 22:49:08.091050 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.091018    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cjc98"]
Oct 29 22:49:08.091234 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:49:08.091214    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cxtg7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-cjc98" podUID="beb265a7-e30c-48a1-adb9-f85d04d4604f"
Oct 29 22:49:08.114557 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.114525    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtg7\" (UniqueName: \"kubernetes.io/projected/beb265a7-e30c-48a1-adb9-f85d04d4604f-kube-api-access-cxtg7\") pod \"authorino-6f6944bcdd-cjc98\" (UID: \"beb265a7-e30c-48a1-adb9-f85d04d4604f\") " pod="kuadrant-system/authorino-6f6944bcdd-cjc98"
Oct 29 22:49:08.118391 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.118368    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-jwk8h"]
Oct 29 22:49:08.121551 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.121536    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.133339 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.133320    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtg7\" (UniqueName: \"kubernetes.io/projected/beb265a7-e30c-48a1-adb9-f85d04d4604f-kube-api-access-cxtg7\") pod \"authorino-6f6944bcdd-cjc98\" (UID: \"beb265a7-e30c-48a1-adb9-f85d04d4604f\") " pod="kuadrant-system/authorino-6f6944bcdd-cjc98"
Oct 29 22:49:08.139206 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.139185    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 29 22:49:08.140129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.140110    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-jwk8h"]
Oct 29 22:49:08.215136 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.215036    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9cc\" (UniqueName: \"kubernetes.io/projected/eb520bfb-e67d-43c9-8334-c7cabe617ef1-kube-api-access-2r9cc\") pod \"authorino-6889cbd8c4-jwk8h\" (UID: \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\") " pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.215136 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.215081    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb520bfb-e67d-43c9-8334-c7cabe617ef1-tls-cert\") pod \"authorino-6889cbd8c4-jwk8h\" (UID: \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\") " pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.249576 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.249549    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-jwk8h"]
Oct 29 22:49:08.249725 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:49:08.249708    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2r9cc tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-jwk8h" podUID="eb520bfb-e67d-43c9-8334-c7cabe617ef1"
Oct 29 22:49:08.276851 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.276808    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cbdcf967d-5kmd7"]
Oct 29 22:49:08.280019 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.279999    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:49:08.297073 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.297045    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbdcf967d-5kmd7"]
Oct 29 22:49:08.316260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.316239    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb520bfb-e67d-43c9-8334-c7cabe617ef1-tls-cert\") pod \"authorino-6889cbd8c4-jwk8h\" (UID: \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\") " pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.316382 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.316277    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32783423-c605-468c-8bde-cdd56b0f6539-tls-cert\") pod \"authorino-cbdcf967d-5kmd7\" (UID: \"32783423-c605-468c-8bde-cdd56b0f6539\") " pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:49:08.316382 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.316297    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9868t\" (UniqueName: \"kubernetes.io/projected/32783423-c605-468c-8bde-cdd56b0f6539-kube-api-access-9868t\") pod \"authorino-cbdcf967d-5kmd7\" (UID: \"32783423-c605-468c-8bde-cdd56b0f6539\") " pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:49:08.316382 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.316331    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9cc\" (UniqueName: \"kubernetes.io/projected/eb520bfb-e67d-43c9-8334-c7cabe617ef1-kube-api-access-2r9cc\") pod \"authorino-6889cbd8c4-jwk8h\" (UID: \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\") " pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.319037 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.319011    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb520bfb-e67d-43c9-8334-c7cabe617ef1-tls-cert\") pod \"authorino-6889cbd8c4-jwk8h\" (UID: \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\") " pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.327494 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.327476    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.327581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.327481    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cjc98"
Oct 29 22:49:08.334144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.334124    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.335448 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.335426    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9cc\" (UniqueName: \"kubernetes.io/projected/eb520bfb-e67d-43c9-8334-c7cabe617ef1-kube-api-access-2r9cc\") pod \"authorino-6889cbd8c4-jwk8h\" (UID: \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\") " pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:08.336700 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.336687    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cjc98"
Oct 29 22:49:08.417172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.417142    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb265a7-e30c-48a1-adb9-f85d04d4604f-kube-api-access-cxtg7\" (UniqueName: \"kubernetes.io/projected/beb265a7-e30c-48a1-adb9-f85d04d4604f-kube-api-access-cxtg7\") pod \"beb265a7-e30c-48a1-adb9-f85d04d4604f\" (UID: \"beb265a7-e30c-48a1-adb9-f85d04d4604f\") "
Oct 29 22:49:08.417172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.417177    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb520bfb-e67d-43c9-8334-c7cabe617ef1-kube-api-access-2r9cc\" (UniqueName: \"kubernetes.io/projected/eb520bfb-e67d-43c9-8334-c7cabe617ef1-kube-api-access-2r9cc\") pod \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\" (UID: \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\") "
Oct 29 22:49:08.417451 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.417314    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb520bfb-e67d-43c9-8334-c7cabe617ef1-tls-cert\" (UniqueName: \"kubernetes.io/secret/eb520bfb-e67d-43c9-8334-c7cabe617ef1-tls-cert\") pod \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\" (UID: \"eb520bfb-e67d-43c9-8334-c7cabe617ef1\") "
Oct 29 22:49:08.417497 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.417477    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32783423-c605-468c-8bde-cdd56b0f6539-tls-cert\") pod \"authorino-cbdcf967d-5kmd7\" (UID: \"32783423-c605-468c-8bde-cdd56b0f6539\") " pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:49:08.417531 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.417517    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9868t\" (UniqueName: \"kubernetes.io/projected/32783423-c605-468c-8bde-cdd56b0f6539-kube-api-access-9868t\") pod \"authorino-cbdcf967d-5kmd7\" (UID: \"32783423-c605-468c-8bde-cdd56b0f6539\") " pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:49:08.419411 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.419377    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb520bfb-e67d-43c9-8334-c7cabe617ef1-tls-cert" pod "eb520bfb-e67d-43c9-8334-c7cabe617ef1" (UID: "eb520bfb-e67d-43c9-8334-c7cabe617ef1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:08.419533 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.419425    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb520bfb-e67d-43c9-8334-c7cabe617ef1-kube-api-access-2r9cc" pod "eb520bfb-e67d-43c9-8334-c7cabe617ef1" (UID: "eb520bfb-e67d-43c9-8334-c7cabe617ef1"). InnerVolumeSpecName "kube-api-access-2r9cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:08.419533 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.419451    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb265a7-e30c-48a1-adb9-f85d04d4604f-kube-api-access-cxtg7" pod "beb265a7-e30c-48a1-adb9-f85d04d4604f" (UID: "beb265a7-e30c-48a1-adb9-f85d04d4604f"). InnerVolumeSpecName "kube-api-access-cxtg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:08.419817 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.419801    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32783423-c605-468c-8bde-cdd56b0f6539-tls-cert\") pod \"authorino-cbdcf967d-5kmd7\" (UID: \"32783423-c605-468c-8bde-cdd56b0f6539\") " pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:49:08.436749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.436718    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9868t\" (UniqueName: \"kubernetes.io/projected/32783423-c605-468c-8bde-cdd56b0f6539-kube-api-access-9868t\") pod \"authorino-cbdcf967d-5kmd7\" (UID: \"32783423-c605-468c-8bde-cdd56b0f6539\") " pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:49:08.518170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.518097    4742 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb520bfb-e67d-43c9-8334-c7cabe617ef1-tls-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:49:08.518170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.518124    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxtg7\" (UniqueName: \"kubernetes.io/projected/beb265a7-e30c-48a1-adb9-f85d04d4604f-kube-api-access-cxtg7\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:49:08.518170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.518135    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r9cc\" (UniqueName: \"kubernetes.io/projected/eb520bfb-e67d-43c9-8334-c7cabe617ef1-kube-api-access-2r9cc\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:49:08.591098 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.591047    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:49:08.878920 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:08.878888    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbdcf967d-5kmd7"]
Oct 29 22:49:08.882138 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:49:08.882080    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32783423_c605_468c_8bde_cdd56b0f6539.slice/crio-469f248a7db08687afc6861c30662b2b7510fe9f9e11d32bbf3e82312fe31f50 WatchSource:0}: Error finding container 469f248a7db08687afc6861c30662b2b7510fe9f9e11d32bbf3e82312fe31f50: Status 404 returned error can't find the container with id 469f248a7db08687afc6861c30662b2b7510fe9f9e11d32bbf3e82312fe31f50
Oct 29 22:49:09.331856 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.331829    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cjc98"
Oct 29 22:49:09.332333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.331856    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbdcf967d-5kmd7" event={"ID":"32783423-c605-468c-8bde-cdd56b0f6539","Type":"ContainerStarted","Data":"44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9"}
Oct 29 22:49:09.332333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.331891    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbdcf967d-5kmd7" event={"ID":"32783423-c605-468c-8bde-cdd56b0f6539","Type":"ContainerStarted","Data":"469f248a7db08687afc6861c30662b2b7510fe9f9e11d32bbf3e82312fe31f50"}
Oct 29 22:49:09.332333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.332013    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-jwk8h"
Oct 29 22:49:09.383381 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.383329    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-cbdcf967d-5kmd7" podStartSLOduration=1.037619368 podStartE2EDuration="1.38331431s" totalImagesPullingTime="345.694942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:09.382526017 +0000 UTC m=+406.523267606" watchObservedRunningTime="2026-10-29 22:49:09.38331431 +0000 UTC m=+406.524055894"
Oct 29 22:49:09.409777 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.409743    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qrss"]
Oct 29 22:49:09.409949 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.409917    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss" podUID="05c6cc3c-835b-4aff-9613-65ec7b1bddd5" containerName="authorino" containerID="cri-o://7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3" gracePeriod=30
Oct 29 22:49:09.444696 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.444662    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-jwk8h"]
Oct 29 22:49:09.453886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.453857    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-jwk8h"]
Oct 29 22:49:09.507204 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.507131    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cjc98"]
Oct 29 22:49:09.509156 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.509133    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cjc98"]
Oct 29 22:49:09.639480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.639457    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss"
Oct 29 22:49:09.727358 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.727333    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c6cc3c-835b-4aff-9613-65ec7b1bddd5-kube-api-access-zs5h5\" (UniqueName: \"kubernetes.io/projected/05c6cc3c-835b-4aff-9613-65ec7b1bddd5-kube-api-access-zs5h5\") pod \"05c6cc3c-835b-4aff-9613-65ec7b1bddd5\" (UID: \"05c6cc3c-835b-4aff-9613-65ec7b1bddd5\") "
Oct 29 22:49:09.729321 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.729295    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c6cc3c-835b-4aff-9613-65ec7b1bddd5-kube-api-access-zs5h5" pod "05c6cc3c-835b-4aff-9613-65ec7b1bddd5" (UID: "05c6cc3c-835b-4aff-9613-65ec7b1bddd5"). InnerVolumeSpecName "kube-api-access-zs5h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:09.828432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:09.828362    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs5h5\" (UniqueName: \"kubernetes.io/projected/05c6cc3c-835b-4aff-9613-65ec7b1bddd5-kube-api-access-zs5h5\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:49:10.336259 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.336225    4742 generic.go:356] "Generic (PLEG): container finished" podID="05c6cc3c-835b-4aff-9613-65ec7b1bddd5" containerID="7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3" exitCode=0
Oct 29 22:49:10.336720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.336265    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss"
Oct 29 22:49:10.336720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.336311    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss" event={"ID":"05c6cc3c-835b-4aff-9613-65ec7b1bddd5","Type":"ContainerDied","Data":"7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3"}
Oct 29 22:49:10.336720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.336352    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4qrss" event={"ID":"05c6cc3c-835b-4aff-9613-65ec7b1bddd5","Type":"ContainerDied","Data":"256e9de17b062ce6e34067a990aef5bbd8b76885b618806ec1b30a6899965c3e"}
Oct 29 22:49:10.336720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.336372    4742 scope.go:122] "RemoveContainer" containerID="7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3"
Oct 29 22:49:10.343944 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.343919    4742 scope.go:122] "RemoveContainer" containerID="7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3"
Oct 29 22:49:10.344275 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:49:10.344250    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3\": container with ID starting with 7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3 not found: ID does not exist" containerID="7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3"
Oct 29 22:49:10.344368 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.344287    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3"} err="failed to get container status \"7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3\": rpc error: code = NotFound desc = could not find container \"7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3\": container with ID starting with 7214f3759d525c142d2167b6181094afb38d695ba187bc7ef76ff74112c717f3 not found: ID does not exist"
Oct 29 22:49:10.390949 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.390887    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qrss"]
Oct 29 22:49:10.397117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.397080    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4qrss"]
Oct 29 22:49:10.992203 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.992169    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c6cc3c-835b-4aff-9613-65ec7b1bddd5" path="/var/lib/kubelet/pods/05c6cc3c-835b-4aff-9613-65ec7b1bddd5/volumes"
Oct 29 22:49:10.992493 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.992480    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb265a7-e30c-48a1-adb9-f85d04d4604f" path="/var/lib/kubelet/pods/beb265a7-e30c-48a1-adb9-f85d04d4604f/volumes"
Oct 29 22:49:10.992696 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:49:10.992686    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb520bfb-e67d-43c9-8334-c7cabe617ef1" path="/var/lib/kubelet/pods/eb520bfb-e67d-43c9-8334-c7cabe617ef1/volumes"
Oct 29 22:49:23.121927 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:49:23.121890    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:49:53.121934 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:49:53.121897    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:50:23.121440 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:50:23.121408    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:50:34.295390 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.295358    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-d5p6h-jwq9n"]
Oct 29 22:50:34.295843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.295614    4742 cpu_manager.go:414] "containerMap: removing container" podUID="05c6cc3c-835b-4aff-9613-65ec7b1bddd5" containerName="authorino"
Oct 29 22:50:34.295843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.295624    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c6cc3c-835b-4aff-9613-65ec7b1bddd5" containerName="authorino"
Oct 29 22:50:34.295843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.295672    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c6cc3c-835b-4aff-9613-65ec7b1bddd5" containerName="authorino"
Oct 29 22:50:34.298548 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.298529    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.324977 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.324956    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rhrpc\""
Oct 29 22:50:34.325110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.325079    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 29 22:50:34.350042 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.350000    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 29 22:50:34.350930 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.350907    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-d5p6h-jwq9n"]
Oct 29 22:50:34.461454 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.461421    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3b8b029e-e8fd-4ef5-a5d7-f58423271966-archives-path\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.461618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.461460    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-service-ca-bundle\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.461618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.461479    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-trusted-ca-bundle\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.461618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.461577    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb995\" (UniqueName: \"kubernetes.io/projected/3b8b029e-e8fd-4ef5-a5d7-f58423271966-kube-api-access-jb995\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.562098 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.562016    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-trusted-ca-bundle\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.562098 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.562066    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb995\" (UniqueName: \"kubernetes.io/projected/3b8b029e-e8fd-4ef5-a5d7-f58423271966-kube-api-access-jb995\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.562270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.562125    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3b8b029e-e8fd-4ef5-a5d7-f58423271966-archives-path\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.562270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.562148    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-service-ca-bundle\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.562539 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.562518    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3b8b029e-e8fd-4ef5-a5d7-f58423271966-archives-path\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.562711 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.562696    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-service-ca-bundle\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.562777 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.562759    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-trusted-ca-bundle\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.582021 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.582003    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb995\" (UniqueName: \"kubernetes.io/projected/3b8b029e-e8fd-4ef5-a5d7-f58423271966-kube-api-access-jb995\") pod \"periodic-gathering-d5p6h-jwq9n\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") " pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.608856 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.608824    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:50:34.896955 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.896925    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-d5p6h-jwq9n"]
Oct 29 22:50:34.900148 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:50:34.900119    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8b029e_e8fd_4ef5_a5d7_f58423271966.slice/crio-74f763e38f945203a39cde5845cc15eb554cc5f12e304f41f558de410eb6fb9a WatchSource:0}: Error finding container 74f763e38f945203a39cde5845cc15eb554cc5f12e304f41f558de410eb6fb9a: Status 404 returned error can't find the container with id 74f763e38f945203a39cde5845cc15eb554cc5f12e304f41f558de410eb6fb9a
Oct 29 22:50:34.902170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:34.902155    4742 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:50:35.627404 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:35.627361    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n" event={"ID":"3b8b029e-e8fd-4ef5-a5d7-f58423271966","Type":"ContainerStarted","Data":"74f763e38f945203a39cde5845cc15eb554cc5f12e304f41f558de410eb6fb9a"}
Oct 29 22:50:37.635480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:37.635450    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n" event={"ID":"3b8b029e-e8fd-4ef5-a5d7-f58423271966","Type":"ContainerStarted","Data":"961ddc65369df0c42920f576c4f154b5834eb3020f2933c1bbafad0e21fb74f6"}
Oct 29 22:50:37.673603 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:37.673556    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n" podStartSLOduration=1.8974488630000002 podStartE2EDuration="3.673542739s" totalImagesPullingTime="1.776093876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:37.672653765 +0000 UTC m=+494.813395351" watchObservedRunningTime="2026-10-29 22:50:37.673542739 +0000 UTC m=+494.814284323"
Oct 29 22:50:53.121925 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:50:53.121882    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:50:56.932941 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:56.932908    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"]
Oct 29 22:50:56.936549 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:56.936528    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:56.964545 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:56.964522    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pzhxp\""
Oct 29 22:50:56.964684 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:56.964569    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 22:50:56.964684 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:56.964593    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 22:50:56.964831 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:56.964817    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 29 22:50:56.965721 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:56.965701    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"]
Oct 29 22:50:57.006198 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.006172    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.006198 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.006202    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wjv\" (UniqueName: \"kubernetes.io/projected/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-kube-api-access-27wjv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.006424 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.006231    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.006485 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.006448    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.006485 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.006475    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.006597 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.006508    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.106887 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.106852    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.106887 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.106888    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.107078 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.106925    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.107078 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.106952    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wjv\" (UniqueName: \"kubernetes.io/projected/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-kube-api-access-27wjv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.107078 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.106986    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.107232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.107124    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.107443 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.107411    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.107601 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.107444    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.107601 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.107504    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.109167 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.109145    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.109449 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.109431    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.127530 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.127499    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wjv\" (UniqueName: \"kubernetes.io/projected/9d1ef9b7-cf83-4679-bd26-947a41a6cc71-kube-api-access-27wjv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj\" (UID: \"9d1ef9b7-cf83-4679-bd26-947a41a6cc71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.252662 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.252576    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:57.546406 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.546375    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"]
Oct 29 22:50:57.548365 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:50:57.548338    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1ef9b7_cf83_4679_bd26_947a41a6cc71.slice/crio-f626891459e85ac97aa6e53bef06259cc538776eed97fb69c893061e69b40634 WatchSource:0}: Error finding container f626891459e85ac97aa6e53bef06259cc538776eed97fb69c893061e69b40634: Status 404 returned error can't find the container with id f626891459e85ac97aa6e53bef06259cc538776eed97fb69c893061e69b40634
Oct 29 22:50:57.705502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:57.705465    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj" event={"ID":"9d1ef9b7-cf83-4679-bd26-947a41a6cc71","Type":"ContainerStarted","Data":"f626891459e85ac97aa6e53bef06259cc538776eed97fb69c893061e69b40634"}
Oct 29 22:50:58.712070 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:58.712028    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj" event={"ID":"9d1ef9b7-cf83-4679-bd26-947a41a6cc71","Type":"ContainerStarted","Data":"61e46c0836734be168246015e20f9ead46cd011818d6d2fd1d04d84d21620f95"}
Oct 29 22:50:58.753851 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:58.753797    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj" podStartSLOduration=1.68317739 podStartE2EDuration="2.753781503s" totalImagesPullingTime="1.070604113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:58.753037066 +0000 UTC m=+515.893778658" watchObservedRunningTime="2026-10-29 22:50:58.753781503 +0000 UTC m=+515.894523087"
Oct 29 22:50:59.252684 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.252647    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:50:59.886706 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.886672    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"]
Oct 29 22:50:59.890389 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.890371    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:50:59.904602 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.904582    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 29 22:50:59.917560 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.917540    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"]
Oct 29 22:50:59.930897 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.930873    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:50:59.931026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.930921    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:50:59.931026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.930987    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145064cd-519a-4ccf-bdfa-b090c300b9b6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:50:59.931026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.931022    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:50:59.931173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.931119    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:50:59.931173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:50:59.931160    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvqj\" (UniqueName: \"kubernetes.io/projected/145064cd-519a-4ccf-bdfa-b090c300b9b6-kube-api-access-zpvqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.031764 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.031729    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.031764 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.031766    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.032011 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.031787    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145064cd-519a-4ccf-bdfa-b090c300b9b6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.032011 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.031819    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.032011 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.031858    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.032011 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.031921    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvqj\" (UniqueName: \"kubernetes.io/projected/145064cd-519a-4ccf-bdfa-b090c300b9b6-kube-api-access-zpvqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.032257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.032216    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.032321 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.032269    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.032387 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.032324    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.034148 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.034125    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145064cd-519a-4ccf-bdfa-b090c300b9b6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.034439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.034421    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145064cd-519a-4ccf-bdfa-b090c300b9b6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.051480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.051454    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvqj\" (UniqueName: \"kubernetes.io/projected/145064cd-519a-4ccf-bdfa-b090c300b9b6-kube-api-access-zpvqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rpq6f\" (UID: \"145064cd-519a-4ccf-bdfa-b090c300b9b6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.202939 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.202860    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:00.493555 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.493481    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"]
Oct 29 22:51:00.496493 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:51:00.496465    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145064cd_519a_4ccf_bdfa_b090c300b9b6.slice/crio-3f04328f46824415e3adf6c28d95523d4997f4d6c5c7d1df73b95fc8ade44bb0 WatchSource:0}: Error finding container 3f04328f46824415e3adf6c28d95523d4997f4d6c5c7d1df73b95fc8ade44bb0: Status 404 returned error can't find the container with id 3f04328f46824415e3adf6c28d95523d4997f4d6c5c7d1df73b95fc8ade44bb0
Oct 29 22:51:00.720445 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:00.720419    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f" event={"ID":"145064cd-519a-4ccf-bdfa-b090c300b9b6","Type":"ContainerStarted","Data":"3f04328f46824415e3adf6c28d95523d4997f4d6c5c7d1df73b95fc8ade44bb0"}
Oct 29 22:51:01.735204 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:01.735159    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f" event={"ID":"145064cd-519a-4ccf-bdfa-b090c300b9b6","Type":"ContainerStarted","Data":"3eacbd6ae4a7f338309f9cf7736fab39965636862f5a228f4dbbf2a2b6db1f92"}
Oct 29 22:51:01.773954 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:01.773889    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f" podStartSLOduration=2.594761688 podStartE2EDuration="2.773872733s" totalImagesPullingTime="179.111045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:01.773860592 +0000 UTC m=+518.914602177" watchObservedRunningTime="2026-10-29 22:51:01.773872733 +0000 UTC m=+518.914614318"
Oct 29 22:51:02.203707 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:02.203661    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:03.906515 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.906480    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"]
Oct 29 22:51:03.910685 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.910659    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:03.938247 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.938216    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 29 22:51:03.940100 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.940058    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"]
Oct 29 22:51:03.968985 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.968939    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:03.969199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.969000    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:03.969199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.969092    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56a8bcc-2991-4a16-891e-fa69e277ca6a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:03.969199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.969130    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:03.969199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.969159    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:03.969426 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:03.969210    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvv8w\" (UniqueName: \"kubernetes.io/projected/f56a8bcc-2991-4a16-891e-fa69e277ca6a-kube-api-access-jvv8w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.070249 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.070214    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvv8w\" (UniqueName: \"kubernetes.io/projected/f56a8bcc-2991-4a16-891e-fa69e277ca6a-kube-api-access-jvv8w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.070249 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.070262    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.070442 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.070283    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.070442 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.070314    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56a8bcc-2991-4a16-891e-fa69e277ca6a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.070442 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.070351    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.070442 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.070375    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.070894 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.070878    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.071397 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.071335    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.071544 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.071488    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.072948 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.072922    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56a8bcc-2991-4a16-891e-fa69e277ca6a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.073810 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.073788    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56a8bcc-2991-4a16-891e-fa69e277ca6a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.093758 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.093725    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvv8w\" (UniqueName: \"kubernetes.io/projected/f56a8bcc-2991-4a16-891e-fa69e277ca6a-kube-api-access-jvv8w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-t7bjt\" (UID: \"f56a8bcc-2991-4a16-891e-fa69e277ca6a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.226104 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.226007    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:04.576161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.575185    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"]
Oct 29 22:51:04.591594 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:51:04.591555    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56a8bcc_2991_4a16_891e_fa69e277ca6a.slice/crio-0f723f5d81a9f03eb0252c9200ad21caa1ef2a31784261106864ddc20bf29cfd WatchSource:0}: Error finding container 0f723f5d81a9f03eb0252c9200ad21caa1ef2a31784261106864ddc20bf29cfd: Status 404 returned error can't find the container with id 0f723f5d81a9f03eb0252c9200ad21caa1ef2a31784261106864ddc20bf29cfd
Oct 29 22:51:04.748692 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:04.748659    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt" event={"ID":"f56a8bcc-2991-4a16-891e-fa69e277ca6a","Type":"ContainerStarted","Data":"0f723f5d81a9f03eb0252c9200ad21caa1ef2a31784261106864ddc20bf29cfd"}
Oct 29 22:51:05.755023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:05.754981    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt" event={"ID":"f56a8bcc-2991-4a16-891e-fa69e277ca6a","Type":"ContainerStarted","Data":"4b521d1595562b399ac8d2b80cc5d3a0c8952c0349ea18ee0df1c7cb1d173590"}
Oct 29 22:51:05.798947 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:05.798887    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt" podStartSLOduration=2.591428016 podStartE2EDuration="2.798871929s" totalImagesPullingTime="207.443913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:05.798078483 +0000 UTC m=+522.938820069" watchObservedRunningTime="2026-10-29 22:51:05.798871929 +0000 UTC m=+522.939613512"
Oct 29 22:51:06.226697 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:06.226655    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:07.253684 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:07.253648    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:51:07.270341 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:07.270313    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:51:07.774536 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:07.774511    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj"
Oct 29 22:51:08.792134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.792079    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"]
Oct 29 22:51:08.795204 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.795182    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.811444 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.811419    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 29 22:51:08.823748 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.823718    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"]
Oct 29 22:51:08.827023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.826986    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b833cd5c-a426-4822-bcff-4965be0c17e4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.827276 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.827257    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.827348 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.827292    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.827405 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.827353    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.827405 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.827386    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgv7\" (UniqueName: \"kubernetes.io/projected/b833cd5c-a426-4822-bcff-4965be0c17e4-kube-api-access-bwgv7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.827489 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.827473    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.928610 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.928573    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.928812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.928646    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b833cd5c-a426-4822-bcff-4965be0c17e4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.928812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.928697    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.928812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.928719    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.928812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.928769    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.928812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.928795    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgv7\" (UniqueName: \"kubernetes.io/projected/b833cd5c-a426-4822-bcff-4965be0c17e4-kube-api-access-bwgv7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.929068 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.929050    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.929182 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.929156    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.929231 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.929170    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.931049 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.931018    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b833cd5c-a426-4822-bcff-4965be0c17e4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.931182 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.931053    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b833cd5c-a426-4822-bcff-4965be0c17e4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:08.948565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:08.948538    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgv7\" (UniqueName: \"kubernetes.io/projected/b833cd5c-a426-4822-bcff-4965be0c17e4-kube-api-access-bwgv7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw\" (UID: \"b833cd5c-a426-4822-bcff-4965be0c17e4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:09.110944 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:09.110909    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:09.440657 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:09.440575    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"]
Oct 29 22:51:09.443791 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:51:09.443750    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb833cd5c_a426_4822_bcff_4965be0c17e4.slice/crio-653fbb7485040957b813324284ad61476a2a66616f24f7cb7bb3767f1de69a62 WatchSource:0}: Error finding container 653fbb7485040957b813324284ad61476a2a66616f24f7cb7bb3767f1de69a62: Status 404 returned error can't find the container with id 653fbb7485040957b813324284ad61476a2a66616f24f7cb7bb3767f1de69a62
Oct 29 22:51:09.782947 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:09.782903    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw" event={"ID":"b833cd5c-a426-4822-bcff-4965be0c17e4","Type":"ContainerStarted","Data":"ca0d59a9b48561143fd13beffd070ff2c3cb6d19e413358e260ed151b12ab4cc"}
Oct 29 22:51:09.783150 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:09.782964    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw" event={"ID":"b833cd5c-a426-4822-bcff-4965be0c17e4","Type":"ContainerStarted","Data":"653fbb7485040957b813324284ad61476a2a66616f24f7cb7bb3767f1de69a62"}
Oct 29 22:51:09.844927 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:09.844874    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw" podStartSLOduration=1.642078536 podStartE2EDuration="1.844858036s" totalImagesPullingTime="202.7795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:09.844343343 +0000 UTC m=+526.985084928" watchObservedRunningTime="2026-10-29 22:51:09.844858036 +0000 UTC m=+526.985599646"
Oct 29 22:51:10.111647 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:10.111613    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:10.203649 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:10.203607    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:10.216155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:10.216128    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:10.799001 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:10.798969    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rpq6f"
Oct 29 22:51:11.721229 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.721194    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"]
Oct 29 22:51:11.725762 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.725736    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.742850 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.742819    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 29 22:51:11.755810 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.755783    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"]
Oct 29 22:51:11.758104 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.758052    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.758239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.758110    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.758239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.758185    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.758354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.758243    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b671691-1df4-4fc2-9ba3-eb27464f3553-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.758354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.758275    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.758354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.758294    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th5n9\" (UniqueName: \"kubernetes.io/projected/2b671691-1df4-4fc2-9ba3-eb27464f3553-kube-api-access-th5n9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.859580 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.859537    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.859818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.859802    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.859942 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.859928    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.860120 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.860103    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b671691-1df4-4fc2-9ba3-eb27464f3553-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.860258 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.860242    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.860357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.860343    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th5n9\" (UniqueName: \"kubernetes.io/projected/2b671691-1df4-4fc2-9ba3-eb27464f3553-kube-api-access-th5n9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.861181 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.861158    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.864111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.863399    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.864275 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.864247    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.865220 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.865192    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b671691-1df4-4fc2-9ba3-eb27464f3553-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.869116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.869069    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b671691-1df4-4fc2-9ba3-eb27464f3553-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:11.887589 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:11.887525    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th5n9\" (UniqueName: \"kubernetes.io/projected/2b671691-1df4-4fc2-9ba3-eb27464f3553-kube-api-access-th5n9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7\" (UID: \"2b671691-1df4-4fc2-9ba3-eb27464f3553\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:12.041502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:12.041408    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:12.351823 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:12.351770    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"]
Oct 29 22:51:12.355601 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:51:12.355572    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b671691_1df4_4fc2_9ba3_eb27464f3553.slice/crio-0bcb70f4066eb2a6c57c45cb12775e039fedc249587a167675d5af5a9b04b0bf WatchSource:0}: Error finding container 0bcb70f4066eb2a6c57c45cb12775e039fedc249587a167675d5af5a9b04b0bf: Status 404 returned error can't find the container with id 0bcb70f4066eb2a6c57c45cb12775e039fedc249587a167675d5af5a9b04b0bf
Oct 29 22:51:12.798822 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:12.798788    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" event={"ID":"2b671691-1df4-4fc2-9ba3-eb27464f3553","Type":"ContainerStarted","Data":"9eaebdd65bc86e76d4b33b27bc07c972899b28b839f082f04ba6b4e790867901"}
Oct 29 22:51:12.798822 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:12.798826    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" event={"ID":"2b671691-1df4-4fc2-9ba3-eb27464f3553","Type":"ContainerStarted","Data":"0bcb70f4066eb2a6c57c45cb12775e039fedc249587a167675d5af5a9b04b0bf"}
Oct 29 22:51:12.839715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:12.839662    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podStartSLOduration=1.644938714 podStartE2EDuration="1.839647794s" totalImagesPullingTime="194.70908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:12.838853395 +0000 UTC m=+529.979594978" watchObservedRunningTime="2026-10-29 22:51:12.839647794 +0000 UTC m=+529.980389372"
Oct 29 22:51:13.042034 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:13.041959    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:14.227145 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:14.227111    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:14.241041 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:14.241010    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:14.817715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:14.817688    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt"
Oct 29 22:51:19.111467 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:19.111411    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:19.123805 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:19.123778    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:19.835815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:19.835787    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw"
Oct 29 22:51:22.042578 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:22.042542    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:22.055180 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:22.055156    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:22.848196 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:22.848169    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:51:23.123479 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:51:23.123397    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:51:29.198231 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.198193    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d8467b9b9-5df8n"]
Oct 29 22:51:29.201632 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.201615    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d8467b9b9-5df8n"
Oct 29 22:51:29.219644 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.219621    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d8467b9b9-5df8n"]
Oct 29 22:51:29.311820 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.311766    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9caaa6b-5c3b-4d63-a334-2f558eeff57a-tls-cert\") pod \"authorino-d8467b9b9-5df8n\" (UID: \"e9caaa6b-5c3b-4d63-a334-2f558eeff57a\") " pod="kuadrant-system/authorino-d8467b9b9-5df8n"
Oct 29 22:51:29.311967 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.311897    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7cql\" (UniqueName: \"kubernetes.io/projected/e9caaa6b-5c3b-4d63-a334-2f558eeff57a-kube-api-access-z7cql\") pod \"authorino-d8467b9b9-5df8n\" (UID: \"e9caaa6b-5c3b-4d63-a334-2f558eeff57a\") " pod="kuadrant-system/authorino-d8467b9b9-5df8n"
Oct 29 22:51:29.412780 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.412745    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9caaa6b-5c3b-4d63-a334-2f558eeff57a-tls-cert\") pod \"authorino-d8467b9b9-5df8n\" (UID: \"e9caaa6b-5c3b-4d63-a334-2f558eeff57a\") " pod="kuadrant-system/authorino-d8467b9b9-5df8n"
Oct 29 22:51:29.412939 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.412818    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7cql\" (UniqueName: \"kubernetes.io/projected/e9caaa6b-5c3b-4d63-a334-2f558eeff57a-kube-api-access-z7cql\") pod \"authorino-d8467b9b9-5df8n\" (UID: \"e9caaa6b-5c3b-4d63-a334-2f558eeff57a\") " pod="kuadrant-system/authorino-d8467b9b9-5df8n"
Oct 29 22:51:29.415199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.415173    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9caaa6b-5c3b-4d63-a334-2f558eeff57a-tls-cert\") pod \"authorino-d8467b9b9-5df8n\" (UID: \"e9caaa6b-5c3b-4d63-a334-2f558eeff57a\") " pod="kuadrant-system/authorino-d8467b9b9-5df8n"
Oct 29 22:51:29.431513 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.431488    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7cql\" (UniqueName: \"kubernetes.io/projected/e9caaa6b-5c3b-4d63-a334-2f558eeff57a-kube-api-access-z7cql\") pod \"authorino-d8467b9b9-5df8n\" (UID: \"e9caaa6b-5c3b-4d63-a334-2f558eeff57a\") " pod="kuadrant-system/authorino-d8467b9b9-5df8n"
Oct 29 22:51:29.512837 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.512748    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d8467b9b9-5df8n"
Oct 29 22:51:29.803860 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.803792    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d8467b9b9-5df8n"]
Oct 29 22:51:29.866065 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:29.866031    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d8467b9b9-5df8n" event={"ID":"e9caaa6b-5c3b-4d63-a334-2f558eeff57a","Type":"ContainerStarted","Data":"3dc7dd09a929c86f7e805ce5301f57e754f917ccf43c8751f33fe2ebba1164ca"}
Oct 29 22:51:30.872024 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:30.871983    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d8467b9b9-5df8n" event={"ID":"e9caaa6b-5c3b-4d63-a334-2f558eeff57a","Type":"ContainerStarted","Data":"ec28ef5b2cbe296c9ee879c8894d5f52fcace38e3212edbf05fa84b2d4818ff3"}
Oct 29 22:51:30.911914 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:30.911847    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d8467b9b9-5df8n" podStartSLOduration=1.397368916 podStartE2EDuration="1.911826293s" totalImagesPullingTime="514.457377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:30.909586449 +0000 UTC m=+548.050328034" watchObservedRunningTime="2026-10-29 22:51:30.911826293 +0000 UTC m=+548.052567879"
Oct 29 22:51:30.939744 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:30.939706    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cbdcf967d-5kmd7"]
Oct 29 22:51:30.939956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:30.939928    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-cbdcf967d-5kmd7" podUID="32783423-c605-468c-8bde-cdd56b0f6539" containerName="authorino" containerID="cri-o://44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9" gracePeriod=30
Oct 29 22:51:31.200192 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.200165    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:51:31.332418 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.332388    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32783423-c605-468c-8bde-cdd56b0f6539-kube-api-access-9868t\" (UniqueName: \"kubernetes.io/projected/32783423-c605-468c-8bde-cdd56b0f6539-kube-api-access-9868t\") pod \"32783423-c605-468c-8bde-cdd56b0f6539\" (UID: \"32783423-c605-468c-8bde-cdd56b0f6539\") "
Oct 29 22:51:31.332580 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.332489    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32783423-c605-468c-8bde-cdd56b0f6539-tls-cert\" (UniqueName: \"kubernetes.io/secret/32783423-c605-468c-8bde-cdd56b0f6539-tls-cert\") pod \"32783423-c605-468c-8bde-cdd56b0f6539\" (UID: \"32783423-c605-468c-8bde-cdd56b0f6539\") "
Oct 29 22:51:31.334508 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.334473    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32783423-c605-468c-8bde-cdd56b0f6539-kube-api-access-9868t" pod "32783423-c605-468c-8bde-cdd56b0f6539" (UID: "32783423-c605-468c-8bde-cdd56b0f6539"). InnerVolumeSpecName "kube-api-access-9868t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:31.341611 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.341587    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32783423-c605-468c-8bde-cdd56b0f6539-tls-cert" pod "32783423-c605-468c-8bde-cdd56b0f6539" (UID: "32783423-c605-468c-8bde-cdd56b0f6539"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:51:31.433429 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.433354    4742 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/32783423-c605-468c-8bde-cdd56b0f6539-tls-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:51:31.433429 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.433389    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9868t\" (UniqueName: \"kubernetes.io/projected/32783423-c605-468c-8bde-cdd56b0f6539-kube-api-access-9868t\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:51:31.880502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.880466    4742 generic.go:356] "Generic (PLEG): container finished" podID="32783423-c605-468c-8bde-cdd56b0f6539" containerID="44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9" exitCode=0
Oct 29 22:51:31.880997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.880521    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbdcf967d-5kmd7"
Oct 29 22:51:31.880997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.880547    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbdcf967d-5kmd7" event={"ID":"32783423-c605-468c-8bde-cdd56b0f6539","Type":"ContainerDied","Data":"44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9"}
Oct 29 22:51:31.880997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.880599    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbdcf967d-5kmd7" event={"ID":"32783423-c605-468c-8bde-cdd56b0f6539","Type":"ContainerDied","Data":"469f248a7db08687afc6861c30662b2b7510fe9f9e11d32bbf3e82312fe31f50"}
Oct 29 22:51:31.880997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.880622    4742 scope.go:122] "RemoveContainer" containerID="44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9"
Oct 29 22:51:31.890762 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.890743    4742 scope.go:122] "RemoveContainer" containerID="44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9"
Oct 29 22:51:31.891124 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:51:31.891097    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9\": container with ID starting with 44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9 not found: ID does not exist" containerID="44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9"
Oct 29 22:51:31.891206 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.891138    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9"} err="failed to get container status \"44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9\": rpc error: code = NotFound desc = could not find container \"44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9\": container with ID starting with 44349bea5da318b74ce57a46fe7ac3240af2264f0314f049f1f9d173216cd9e9 not found: ID does not exist"
Oct 29 22:51:31.934405 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.934375    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cbdcf967d-5kmd7"]
Oct 29 22:51:31.946447 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:31.946413    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cbdcf967d-5kmd7"]
Oct 29 22:51:32.990552 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:32.990522    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32783423-c605-468c-8bde-cdd56b0f6539" path="/var/lib/kubelet/pods/32783423-c605-468c-8bde-cdd56b0f6539/volumes"
Oct 29 22:51:34.964880 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:34.964846    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"]
Oct 29 22:51:34.965292 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:34.965219    4742 cpu_manager.go:414] "containerMap: removing container" podUID="32783423-c605-468c-8bde-cdd56b0f6539" containerName="authorino"
Oct 29 22:51:34.965292 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:34.965231    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32783423-c605-468c-8bde-cdd56b0f6539" containerName="authorino"
Oct 29 22:51:34.965292 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:34.965287    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="32783423-c605-468c-8bde-cdd56b0f6539" containerName="authorino"
Oct 29 22:51:34.974819 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:34.974794    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:34.990571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:34.990550    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"]
Oct 29 22:51:35.024911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.024880    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2d967\""
Oct 29 22:51:35.024911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.024891    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 29 22:51:35.077117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.077071    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5p66\" (UniqueName: \"kubernetes.io/projected/6ad493a0-72fe-4341-9ae5-1dd71d2d811f-kube-api-access-p5p66\") pod \"payload-pre-processing-7b98bb678c-tlj45\" (UID: \"6ad493a0-72fe-4341-9ae5-1dd71d2d811f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:35.077117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.077126    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad493a0-72fe-4341-9ae5-1dd71d2d811f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tlj45\" (UID: \"6ad493a0-72fe-4341-9ae5-1dd71d2d811f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:35.178009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.177976    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5p66\" (UniqueName: \"kubernetes.io/projected/6ad493a0-72fe-4341-9ae5-1dd71d2d811f-kube-api-access-p5p66\") pod \"payload-pre-processing-7b98bb678c-tlj45\" (UID: \"6ad493a0-72fe-4341-9ae5-1dd71d2d811f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:35.178009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.178007    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad493a0-72fe-4341-9ae5-1dd71d2d811f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tlj45\" (UID: \"6ad493a0-72fe-4341-9ae5-1dd71d2d811f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:35.178699 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.178679    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad493a0-72fe-4341-9ae5-1dd71d2d811f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tlj45\" (UID: \"6ad493a0-72fe-4341-9ae5-1dd71d2d811f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:35.197254 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.197229    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5p66\" (UniqueName: \"kubernetes.io/projected/6ad493a0-72fe-4341-9ae5-1dd71d2d811f-kube-api-access-p5p66\") pod \"payload-pre-processing-7b98bb678c-tlj45\" (UID: \"6ad493a0-72fe-4341-9ae5-1dd71d2d811f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:35.288883 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.288817    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:35.580372 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.580318    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"]
Oct 29 22:51:35.582659 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:51:35.582632    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad493a0_72fe_4341_9ae5_1dd71d2d811f.slice/crio-727b65ced8cf7f5b7c47bdea5a0da5f981e26eb46b02b3b21c2ad8851d837c6f WatchSource:0}: Error finding container 727b65ced8cf7f5b7c47bdea5a0da5f981e26eb46b02b3b21c2ad8851d837c6f: Status 404 returned error can't find the container with id 727b65ced8cf7f5b7c47bdea5a0da5f981e26eb46b02b3b21c2ad8851d837c6f
Oct 29 22:51:35.899346 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:35.899311    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45" event={"ID":"6ad493a0-72fe-4341-9ae5-1dd71d2d811f","Type":"ContainerStarted","Data":"727b65ced8cf7f5b7c47bdea5a0da5f981e26eb46b02b3b21c2ad8851d837c6f"}
Oct 29 22:51:37.828911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.828826    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"]
Oct 29 22:51:37.832292 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.832266    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:37.851070 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.851047    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"]
Oct 29 22:51:37.853981 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.853966    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:37.860443 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.860422    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z74sk\""
Oct 29 22:51:37.862672 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.862646    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"]
Oct 29 22:51:37.886654 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.886632    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"]
Oct 29 22:51:37.908644 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.908617    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45" event={"ID":"6ad493a0-72fe-4341-9ae5-1dd71d2d811f","Type":"ContainerStarted","Data":"cbccc90b6fde297604949180764b79189a25dc450070e0a2757e8197fc83d814"}
Oct 29 22:51:37.908784 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.908738    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:37.960625 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:37.960542    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45" podStartSLOduration=2.141925055 podStartE2EDuration="3.960525042s" totalImagesPullingTime="1.818599987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:37.958687536 +0000 UTC m=+555.099429102" watchObservedRunningTime="2026-10-29 22:51:37.960525042 +0000 UTC m=+555.101266620"
Oct 29 22:51:38.004645 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.004611    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.004813 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.004693    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07991432-059a-40a2-9295-8d113f364302-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.004813 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.004729    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.004921 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.004902    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.004967 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.004949    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.005023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005010    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.005060 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005036    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.005115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005078    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.005152 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005130    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07991432-059a-40a2-9295-8d113f364302-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.005209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005195    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.005273 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005257    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.005353 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005326    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2rc\" (UniqueName: \"kubernetes.io/projected/07991432-059a-40a2-9295-8d113f364302-kube-api-access-hd2rc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.005472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005376    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07991432-059a-40a2-9295-8d113f364302-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.005472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005405    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.005472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005456    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.005582 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005477    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.005582 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005522    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.005582 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.005546    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbgx\" (UniqueName: \"kubernetes.io/projected/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-kube-api-access-6sbgx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.105939 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.105907    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07991432-059a-40a2-9295-8d113f364302-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.105948    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.105964    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.105978    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.105998    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106135    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbgx\" (UniqueName: \"kubernetes.io/projected/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-kube-api-access-6sbgx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106178    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106228    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07991432-059a-40a2-9295-8d113f364302-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106252    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106279    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106334    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106374    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106399    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106406    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106431    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106706    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.106815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106715    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106433    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106761    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07991432-059a-40a2-9295-8d113f364302-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106785    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.106815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106805    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.107210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106833    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.107210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106869    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2rc\" (UniqueName: \"kubernetes.io/projected/07991432-059a-40a2-9295-8d113f364302-kube-api-access-hd2rc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.107210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106964    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07991432-059a-40a2-9295-8d113f364302-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.107210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.106963    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.107210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.107157    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.107467 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.107213    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.107467 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.107297    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.108475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.108453    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.108825 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.108794    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07991432-059a-40a2-9295-8d113f364302-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.109258 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.109241    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07991432-059a-40a2-9295-8d113f364302-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.109321 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.109247    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.129742 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.129715    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbgx\" (UniqueName: \"kubernetes.io/projected/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-kube-api-access-6sbgx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.129827 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.129770    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ac9a2a-6abb-4ddb-aa72-b38f589e13b1-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fbvvd\" (UID: \"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.129984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.129966    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07991432-059a-40a2-9295-8d113f364302-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.130025 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.130006    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2rc\" (UniqueName: \"kubernetes.io/projected/07991432-059a-40a2-9295-8d113f364302-kube-api-access-hd2rc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-sk7lx\" (UID: \"07991432-059a-40a2-9295-8d113f364302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.147901 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.147884    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:38.167038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.167011    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:38.467110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.467055    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"]
Oct 29 22:51:38.469794 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:51:38.469767    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ac9a2a_6abb_4ddb_aa72_b38f589e13b1.slice/crio-8b2c5f202e60a6de4619606f9dd0c2d6c3fb5724f3db5e3df09b65a8958986ed WatchSource:0}: Error finding container 8b2c5f202e60a6de4619606f9dd0c2d6c3fb5724f3db5e3df09b65a8958986ed: Status 404 returned error can't find the container with id 8b2c5f202e60a6de4619606f9dd0c2d6c3fb5724f3db5e3df09b65a8958986ed
Oct 29 22:51:38.470505 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.470485    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"]
Oct 29 22:51:38.472668 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.472640    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:38.472777 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.472695    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:38.472777 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.472724    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:38.484140 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:51:38.484076    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07991432_059a_40a2_9295_8d113f364302.slice/crio-618b9e7dc256a598d25d975d53ccffde0e78257ab13087764133e677a4235664 WatchSource:0}: Error finding container 618b9e7dc256a598d25d975d53ccffde0e78257ab13087764133e677a4235664: Status 404 returned error can't find the container with id 618b9e7dc256a598d25d975d53ccffde0e78257ab13087764133e677a4235664
Oct 29 22:51:38.486852 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.486819    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:38.486956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.486891    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:38.486956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.486937    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:38.928943 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.928898    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" event={"ID":"07991432-059a-40a2-9295-8d113f364302","Type":"ContainerStarted","Data":"87009198d0e7c2af2a756290b94160af33acfb2091ffd391712a0f808cbd45b4"}
Oct 29 22:51:38.929613 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.929587    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" event={"ID":"07991432-059a-40a2-9295-8d113f364302","Type":"ContainerStarted","Data":"618b9e7dc256a598d25d975d53ccffde0e78257ab13087764133e677a4235664"}
Oct 29 22:51:38.933630 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.933606    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" event={"ID":"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1","Type":"ContainerStarted","Data":"9ea43fa62b609de1e9d088e18239a11a0406e434d8c19ed46496892cb8a4c13f"}
Oct 29 22:51:38.933821 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.933806    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" event={"ID":"25ac9a2a-6abb-4ddb-aa72-b38f589e13b1","Type":"ContainerStarted","Data":"8b2c5f202e60a6de4619606f9dd0c2d6c3fb5724f3db5e3df09b65a8958986ed"}
Oct 29 22:51:38.993107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:38.988369    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podStartSLOduration=1.9883489939999999 podStartE2EDuration="1.988348994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:38.98438718 +0000 UTC m=+556.125128780" watchObservedRunningTime="2026-10-29 22:51:38.988348994 +0000 UTC m=+556.129090579"
Oct 29 22:51:39.028884 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:39.028824    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podStartSLOduration=2.028793811 podStartE2EDuration="2.028793811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:39.028567076 +0000 UTC m=+556.169308660" watchObservedRunningTime="2026-10-29 22:51:39.028793811 +0000 UTC m=+556.169535395"
Oct 29 22:51:39.148759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:39.148720    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:39.150112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:39.150067    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:39.150238 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:39.150134    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:51:39.167442 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:39.167406    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:39.174345 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:39.174214    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:39.174345 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:39.174275    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:40.148363 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:40.148328    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:40.148744 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:40.148383    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:51:40.170693 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:40.170658    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:40.170970 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:40.170934    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:41.149126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:41.149031    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:41.149126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:41.149118    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:51:41.176499 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:41.176464    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:41.176770 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:41.176746    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:42.148943 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:42.148906    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:42.149137 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:42.148959    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:51:42.178107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:42.177458    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:42.178107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:42.177512    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:43.148973 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:43.148936    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:43.149153 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:43.149008    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:51:43.172466 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:43.172431    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:43.172712 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:43.172685    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:44.149146 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:44.149107    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:44.149615 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:44.149175    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:51:44.180110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:44.179189    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:44.180110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:44.179255    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:45.148593 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:45.148552    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:45.148818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:45.148604    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 29 22:51:45.167929 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:45.167895    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:45.168402 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:45.167958    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 29 22:51:46.802974 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:46.802928    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:46.803449 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:46.803354    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:46.804369 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:46.804347    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd"
Oct 29 22:51:46.808595 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:46.808572    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:46.809111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:46.809065    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:46.809906 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:46.809881    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx"
Oct 29 22:51:48.937057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:48.937033    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tlj45"
Oct 29 22:51:52.915491 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:52.915447    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"]
Oct 29 22:51:52.920128 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:52.920101    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:52.948456 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:52.948428    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"]
Oct 29 22:51:53.089925 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.089887    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.089925 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.089928    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.090161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.089949    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.090161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.090063    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvs7\" (UniqueName: \"kubernetes.io/projected/87c1ed81-d887-47a9-a02e-04da08ca80b6-kube-api-access-bwvs7\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.090220 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.090184    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.090220 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.090203    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.090283 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.090230    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.090317 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.090287    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.090352 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.090314    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c1ed81-d887-47a9-a02e-04da08ca80b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.122227 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:51:53.122131    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:51:53.190815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.190726    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.190815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.190761    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c1ed81-d887-47a9-a02e-04da08ca80b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.190815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.190813    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191159 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.190838    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191159 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.190862    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191159 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.190895    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvs7\" (UniqueName: \"kubernetes.io/projected/87c1ed81-d887-47a9-a02e-04da08ca80b6-kube-api-access-bwvs7\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191159 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.190966    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191159 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.190989    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191159 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.191027    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191476 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.191195    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191476 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.191389    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191619 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.191598    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191757 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.191739    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c1ed81-d887-47a9-a02e-04da08ca80b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.191797 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.191763    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.193768 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.193743    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.194195 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.194170    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.208951 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.208922    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c1ed81-d887-47a9-a02e-04da08ca80b6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.209762 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.209737    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvs7\" (UniqueName: \"kubernetes.io/projected/87c1ed81-d887-47a9-a02e-04da08ca80b6-kube-api-access-bwvs7\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-c7pw8\" (UID: \"87c1ed81-d887-47a9-a02e-04da08ca80b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.233659 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.233629    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:53.540431 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.540353    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"]
Oct 29 22:51:53.543548 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:51:53.543525    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c1ed81_d887_47a9_a02e_04da08ca80b6.slice/crio-5d44a2b3e6c6bfb46160fa0664c22502419fed8fb382fd62555405644a841031 WatchSource:0}: Error finding container 5d44a2b3e6c6bfb46160fa0664c22502419fed8fb382fd62555405644a841031: Status 404 returned error can't find the container with id 5d44a2b3e6c6bfb46160fa0664c22502419fed8fb382fd62555405644a841031
Oct 29 22:51:53.545542 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.545510    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:53.545622 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.545572    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:53.545622 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.545599    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:51:53.995234 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.995194    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" event={"ID":"87c1ed81-d887-47a9-a02e-04da08ca80b6","Type":"ContainerStarted","Data":"3ee89ca14bf976d6ac5c4cc3df8bffb0c5c6386f8634afe0a4cefed2da277621"}
Oct 29 22:51:53.995663 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:53.995253    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" event={"ID":"87c1ed81-d887-47a9-a02e-04da08ca80b6","Type":"ContainerStarted","Data":"5d44a2b3e6c6bfb46160fa0664c22502419fed8fb382fd62555405644a841031"}
Oct 29 22:51:54.079117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:54.075939    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podStartSLOduration=2.075916172 podStartE2EDuration="2.075916172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:54.047958769 +0000 UTC m=+571.188700353" watchObservedRunningTime="2026-10-29 22:51:54.075916172 +0000 UTC m=+571.216657753"
Oct 29 22:51:54.234514 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:54.234479    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:51:54.235922 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:54.235897    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:51:54.236038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:54.235942    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:51:55.234668 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:55.234634    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:51:55.235056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:55.234715    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:51:56.234501 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:56.234464    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:51:56.234710 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:56.234524    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:51:57.234047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:57.234011    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:51:57.234240 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:57.234066    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:51:58.234719 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:58.234687    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:51:58.235150 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:58.234737    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:51:59.234532 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:59.234491    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:51:59.234741 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:51:59.234554    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:52:00.234131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:00.234072    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:00.234297 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:00.234158    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 22:52:01.742455 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:01.742418    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:52:01.742924 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:01.742841    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:52:01.743543 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:01.743526    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8"
Oct 29 22:52:12.074957 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:12.074861    4742 generic.go:356] "Generic (PLEG): container finished" podID="3b8b029e-e8fd-4ef5-a5d7-f58423271966" containerID="961ddc65369df0c42920f576c4f154b5834eb3020f2933c1bbafad0e21fb74f6" exitCode=0
Oct 29 22:52:12.074957 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:12.074930    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n" event={"ID":"3b8b029e-e8fd-4ef5-a5d7-f58423271966","Type":"ContainerDied","Data":"961ddc65369df0c42920f576c4f154b5834eb3020f2933c1bbafad0e21fb74f6"}
Oct 29 22:52:13.210346 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.210323    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:52:13.264076 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.264048    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8b029e-e8fd-4ef5-a5d7-f58423271966-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3b8b029e-e8fd-4ef5-a5d7-f58423271966-archives-path\") pod \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") "
Oct 29 22:52:13.264241 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.264117    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-trusted-ca-bundle\") pod \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") "
Oct 29 22:52:13.264241 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.264152    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-service-ca-bundle\") pod \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") "
Oct 29 22:52:13.264241 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.264177    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8b029e-e8fd-4ef5-a5d7-f58423271966-kube-api-access-jb995\" (UniqueName: \"kubernetes.io/projected/3b8b029e-e8fd-4ef5-a5d7-f58423271966-kube-api-access-jb995\") pod \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\" (UID: \"3b8b029e-e8fd-4ef5-a5d7-f58423271966\") "
Oct 29 22:52:13.264525 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.264494    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8b029e-e8fd-4ef5-a5d7-f58423271966-archives-path" pod "3b8b029e-e8fd-4ef5-a5d7-f58423271966" (UID: "3b8b029e-e8fd-4ef5-a5d7-f58423271966"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:13.264586 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.264516    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-trusted-ca-bundle" pod "3b8b029e-e8fd-4ef5-a5d7-f58423271966" (UID: "3b8b029e-e8fd-4ef5-a5d7-f58423271966"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:13.264586 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.264520    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-service-ca-bundle" pod "3b8b029e-e8fd-4ef5-a5d7-f58423271966" (UID: "3b8b029e-e8fd-4ef5-a5d7-f58423271966"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:13.266349 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.266325    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8b029e-e8fd-4ef5-a5d7-f58423271966-kube-api-access-jb995" pod "3b8b029e-e8fd-4ef5-a5d7-f58423271966" (UID: "3b8b029e-e8fd-4ef5-a5d7-f58423271966"). InnerVolumeSpecName "kube-api-access-jb995". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:13.365057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.365025    4742 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3b8b029e-e8fd-4ef5-a5d7-f58423271966-archives-path\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:52:13.365057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.365054    4742 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-trusted-ca-bundle\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:52:13.365273 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.365063    4742 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b8b029e-e8fd-4ef5-a5d7-f58423271966-service-ca-bundle\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:52:13.365273 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:13.365073    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb995\" (UniqueName: \"kubernetes.io/projected/3b8b029e-e8fd-4ef5-a5d7-f58423271966-kube-api-access-jb995\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:52:14.084620 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:14.084591    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n"
Oct 29 22:52:14.084620 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:14.084616    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d5p6h-jwq9n" event={"ID":"3b8b029e-e8fd-4ef5-a5d7-f58423271966","Type":"ContainerDied","Data":"74f763e38f945203a39cde5845cc15eb554cc5f12e304f41f558de410eb6fb9a"}
Oct 29 22:52:14.084834 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:14.084657    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74f763e38f945203a39cde5845cc15eb554cc5f12e304f41f558de410eb6fb9a"
Oct 29 22:52:23.122966 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:52:23.122932    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:52:24.802974 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.802923    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"]
Oct 29 22:52:24.804109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.804066    4742 cpu_manager.go:414] "containerMap: removing container" podUID="3b8b029e-e8fd-4ef5-a5d7-f58423271966" containerName="insights-gathering"
Oct 29 22:52:24.804109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.804106    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8b029e-e8fd-4ef5-a5d7-f58423271966" containerName="insights-gathering"
Oct 29 22:52:24.804251 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.804213    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8b029e-e8fd-4ef5-a5d7-f58423271966" containerName="insights-gathering"
Oct 29 22:52:24.808714 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.808694    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.827269 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.827235    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5bf55a-openshift-default-dockercfg-hx7k2\""
Oct 29 22:52:24.840346 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.840320    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"]
Oct 29 22:52:24.864791 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.864759    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-podinfo\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.864791 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.864793    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-credential-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.864969 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.864820    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-certs\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.864969 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.864953    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-data\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.865047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.864981    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9dl\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-kube-api-access-rm9dl\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.865047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.865014    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f7435-1e6b-4e25-af1a-c399471ca25d-istiod-ca-cert\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.865147 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.865076    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-envoy\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.865147 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.865134    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.865219 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.865157    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-token\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966471    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-data\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966502    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9dl\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-kube-api-access-rm9dl\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966562    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f7435-1e6b-4e25-af1a-c399471ca25d-istiod-ca-cert\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966590    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-envoy\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966621    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966652    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-token\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966928 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966751    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-podinfo\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966928 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966782    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-credential-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966928 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966808    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-certs\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.966928 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966919    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-data\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.967124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.966980    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.967404 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.967380    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-certs\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.967529 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.967451    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-credential-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.967824 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.967800    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f7435-1e6b-4e25-af1a-c399471ca25d-istiod-ca-cert\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.968962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.968935    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-envoy\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.969299 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.969281    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-podinfo\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.994016 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.993992    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9dl\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-kube-api-access-rm9dl\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:24.994816 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:24.994788    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-token\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:25.095504 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.095473    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"]
Oct 29 22:52:25.100104 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.100067    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.119661 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.119619    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ffba57-openshift-default-dockercfg-5l7zt\""
Oct 29 22:52:25.122759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.122038    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"]
Oct 29 22:52:25.123480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.123456    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:25.169053 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169016    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.169209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169116    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.169209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169159    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.169209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169189    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.169327 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169215    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkg65\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-kube-api-access-qkg65\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.169327 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169284    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.169401 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169326    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.169401 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169354    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.169473 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.169407    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270147 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270115    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270322 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270165    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270322 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270191    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270322 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270237    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270322 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270278    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270550 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270323    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270550 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270362    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270550 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270390    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270550 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270415    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkg65\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-kube-api-access-qkg65\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270824 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270604    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270876 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270829    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.270876 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270843    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.271005 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.270980    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.271154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.271135    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.272572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.272548    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.272816 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.272789    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.289741 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.289713    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkg65\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-kube-api-access-qkg65\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.289911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.289886    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.402146 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.402044    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"]
Oct 29 22:52:25.406821 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.406795    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.416561 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.416542    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:25.431614 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.431593    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-07eef2-openshift-default-dockercfg-5l9dg\""
Oct 29 22:52:25.434214 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.434173    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"]
Oct 29 22:52:25.452380 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.452202    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"]
Oct 29 22:52:25.454623 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:25.454592    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57f7435_1e6b_4e25_af1a_c399471ca25d.slice/crio-fc1b3585fa6c692af22b0dbe4d7f02d0c48f17803d0575a860acb67ba5c620e8 WatchSource:0}: Error finding container fc1b3585fa6c692af22b0dbe4d7f02d0c48f17803d0575a860acb67ba5c620e8: Status 404 returned error can't find the container with id fc1b3585fa6c692af22b0dbe4d7f02d0c48f17803d0575a860acb67ba5c620e8
Oct 29 22:52:25.458746 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.458302    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:25.458746 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.458368    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:25.458746 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.458397    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:25.472543 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.472508    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.474296 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.474272    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.474350 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.474326    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.474388 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.474376    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmft4\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-kube-api-access-vmft4\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.474454 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.474439    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091431d4-cb1e-46ef-babb-e5f54e589476-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.474501 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.474471    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.474535 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.474500    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.474535 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.474524    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.474598 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.474567    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091431d4-cb1e-46ef-babb-e5f54e589476-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575344 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575297    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575344 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575346    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575629 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575381    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091431d4-cb1e-46ef-babb-e5f54e589476-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575629 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575456    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575629 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575512    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575629 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575549    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575629 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575598    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmft4\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-kube-api-access-vmft4\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575861 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575642    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091431d4-cb1e-46ef-babb-e5f54e589476-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.575861 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575669    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.576007 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.575990    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.576197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.576176    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.576436 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.576417    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.577071 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.577048    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.577839 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.577816    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091431d4-cb1e-46ef-babb-e5f54e589476-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.579440 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.579413    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091431d4-cb1e-46ef-babb-e5f54e589476-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.579816 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.579793    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.599446 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.599421    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmft4\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-kube-api-access-vmft4\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.599784 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.599762    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.721868 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.720969    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"]
Oct 29 22:52:25.721868 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.721071    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:25.723425 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:25.723393    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4a636e_a3b0_4347_8b47_68c6b6dca509.slice/crio-2a4a776f1cded10f1ff214a0df4a7c6684109629094fa2607466eb81e39a05b4 WatchSource:0}: Error finding container 2a4a776f1cded10f1ff214a0df4a7c6684109629094fa2607466eb81e39a05b4: Status 404 returned error can't find the container with id 2a4a776f1cded10f1ff214a0df4a7c6684109629094fa2607466eb81e39a05b4
Oct 29 22:52:25.725920 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.725884    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:25.726010 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.725966    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:25.726060 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:25.726016    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:26.048456 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.048378    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"]
Oct 29 22:52:26.056649 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.056619    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:26.056769 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.056678    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:26.056769 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.056706    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:26.139956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.139924    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" event={"ID":"091431d4-cb1e-46ef-babb-e5f54e589476","Type":"ContainerStarted","Data":"255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40"}
Oct 29 22:52:26.140142 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.139986    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" event={"ID":"091431d4-cb1e-46ef-babb-e5f54e589476","Type":"ContainerStarted","Data":"04d9e7bcdfcd776d751da11e780989ad3582650cb0dff73f73dd0b8f7ecec613"}
Oct 29 22:52:26.142015 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.141988    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" event={"ID":"7c4a636e-a3b0-4347-8b47-68c6b6dca509","Type":"ContainerStarted","Data":"e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294"}
Oct 29 22:52:26.142162 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.142040    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" event={"ID":"7c4a636e-a3b0-4347-8b47-68c6b6dca509","Type":"ContainerStarted","Data":"2a4a776f1cded10f1ff214a0df4a7c6684109629094fa2607466eb81e39a05b4"}
Oct 29 22:52:26.143831 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.143808    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" event={"ID":"e57f7435-1e6b-4e25-af1a-c399471ca25d","Type":"ContainerStarted","Data":"db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82"}
Oct 29 22:52:26.143940 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.143844    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" event={"ID":"e57f7435-1e6b-4e25-af1a-c399471ca25d","Type":"ContainerStarted","Data":"fc1b3585fa6c692af22b0dbe4d7f02d0c48f17803d0575a860acb67ba5c620e8"}
Oct 29 22:52:26.183525 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.183468    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" podStartSLOduration=1.183449609 podStartE2EDuration="1.183449609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:26.182586104 +0000 UTC m=+603.323327692" watchObservedRunningTime="2026-10-29 22:52:26.183449609 +0000 UTC m=+603.324191193"
Oct 29 22:52:26.224000 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.223944    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" podStartSLOduration=1.223928787 podStartE2EDuration="1.223928787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:26.222834012 +0000 UTC m=+603.363575596" watchObservedRunningTime="2026-10-29 22:52:26.223928787 +0000 UTC m=+603.364670371"
Oct 29 22:52:26.266239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.266179    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" podStartSLOduration=2.266164173 podStartE2EDuration="2.266164173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:26.264205496 +0000 UTC m=+603.404947105" watchObservedRunningTime="2026-10-29 22:52:26.266164173 +0000 UTC m=+603.406905792"
Oct 29 22:52:26.417405 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.417370    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:26.422382 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.422357    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:26.721568 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:26.721476    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:27.124484 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:27.124441    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:27.129434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:27.129409    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:27.149129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:27.149102    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:27.149323 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:27.149291    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:27.149719 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:27.149702    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:27.149917 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:27.149905    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:52:27.726242 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:27.726214    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:28.154393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:28.154354    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:28.155192 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:28.155176    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:30.148077 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.148042    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"]
Oct 29 22:52:30.151863 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.151840    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:30.178252 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.178215    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-20f88f\""
Oct 29 22:52:30.179996 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.179948    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"]
Oct 29 22:52:30.187913 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.187879    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-20f88f-dockercfg-hw4mb\""
Oct 29 22:52:30.225547 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.225496    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b61d8c13-0e9f-4f30-be94-cddeb5abb510-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5\" (UID: \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:30.225751 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.225660    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6chm\" (UniqueName: \"kubernetes.io/projected/b61d8c13-0e9f-4f30-be94-cddeb5abb510-kube-api-access-x6chm\") pod \"payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5\" (UID: \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:30.327029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.326989    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6chm\" (UniqueName: \"kubernetes.io/projected/b61d8c13-0e9f-4f30-be94-cddeb5abb510-kube-api-access-x6chm\") pod \"payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5\" (UID: \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:30.327217 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.327167    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b61d8c13-0e9f-4f30-be94-cddeb5abb510-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5\" (UID: \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:30.327863 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.327832    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b61d8c13-0e9f-4f30-be94-cddeb5abb510-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5\" (UID: \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:30.357440 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.357407    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6chm\" (UniqueName: \"kubernetes.io/projected/b61d8c13-0e9f-4f30-be94-cddeb5abb510-kube-api-access-x6chm\") pod \"payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5\" (UID: \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:30.465666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.465572    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:30.763867 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:30.763781    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"]
Oct 29 22:52:30.766816 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:30.766791    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61d8c13_0e9f_4f30_be94_cddeb5abb510.slice/crio-26c8920379efcd29ebc1c5e6de93d1055497b44e3627f97d86cc4bdbd2148def WatchSource:0}: Error finding container 26c8920379efcd29ebc1c5e6de93d1055497b44e3627f97d86cc4bdbd2148def: Status 404 returned error can't find the container with id 26c8920379efcd29ebc1c5e6de93d1055497b44e3627f97d86cc4bdbd2148def
Oct 29 22:52:31.167981 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.167939    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" event={"ID":"b61d8c13-0e9f-4f30-be94-cddeb5abb510","Type":"ContainerStarted","Data":"b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8"}
Oct 29 22:52:31.168410 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.168002    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" event={"ID":"b61d8c13-0e9f-4f30-be94-cddeb5abb510","Type":"ContainerStarted","Data":"26c8920379efcd29ebc1c5e6de93d1055497b44e3627f97d86cc4bdbd2148def"}
Oct 29 22:52:31.168410 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.168041    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:31.211566 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.211515    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" podStartSLOduration=1.2114984739999999 podStartE2EDuration="1.211498474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:31.209946431 +0000 UTC m=+608.350688016" watchObservedRunningTime="2026-10-29 22:52:31.211498474 +0000 UTC m=+608.352240059"
Oct 29 22:52:31.916813 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.916777    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"]
Oct 29 22:52:31.920221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.920205    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:31.937875 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.937850    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-07eef2-dockercfg-jj4wm\""
Oct 29 22:52:31.939932 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.939911    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"]
Oct 29 22:52:31.949675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:31.949652    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-07eef2\""
Oct 29 22:52:32.030278 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.030241    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"]
Oct 29 22:52:32.036067 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.036040    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:32.048180 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.048156    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4td\" (UniqueName: \"kubernetes.io/projected/f952eacf-1713-4dba-890b-ac9838b1636b-kube-api-access-6x4td\") pod \"payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4\" (UID: \"f952eacf-1713-4dba-890b-ac9838b1636b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:32.048180 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.048297    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952eacf-1713-4dba-890b-ac9838b1636b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4\" (UID: \"f952eacf-1713-4dba-890b-ac9838b1636b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:32.059194 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.059166    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"]
Oct 29 22:52:32.149379 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.149342    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4td\" (UniqueName: \"kubernetes.io/projected/f952eacf-1713-4dba-890b-ac9838b1636b-kube-api-access-6x4td\") pod \"payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4\" (UID: \"f952eacf-1713-4dba-890b-ac9838b1636b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:32.149555 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.149401    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr\" (UID: \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:32.149555 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.149445    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952eacf-1713-4dba-890b-ac9838b1636b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4\" (UID: \"f952eacf-1713-4dba-890b-ac9838b1636b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:32.149555 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.149537    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlltx\" (UniqueName: \"kubernetes.io/projected/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-kube-api-access-nlltx\") pod \"payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr\" (UID: \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:32.150220 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.150201    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952eacf-1713-4dba-890b-ac9838b1636b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4\" (UID: \"f952eacf-1713-4dba-890b-ac9838b1636b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:32.173404 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.173343    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4td\" (UniqueName: \"kubernetes.io/projected/f952eacf-1713-4dba-890b-ac9838b1636b-kube-api-access-6x4td\") pod \"payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4\" (UID: \"f952eacf-1713-4dba-890b-ac9838b1636b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:32.233325 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.233285    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:32.250707 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.250678    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr\" (UID: \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:32.250829 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.250768    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlltx\" (UniqueName: \"kubernetes.io/projected/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-kube-api-access-nlltx\") pod \"payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr\" (UID: \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:32.251525 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.251500    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr\" (UID: \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:32.271943 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.271913    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlltx\" (UniqueName: \"kubernetes.io/projected/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-kube-api-access-nlltx\") pod \"payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr\" (UID: \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:32.354843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.354806    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:32.580494 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.580284    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"]
Oct 29 22:52:32.582620 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:32.582593    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf952eacf_1713_4dba_890b_ac9838b1636b.slice/crio-e8f9177ac258f692a581f1d6422ac5e6be0a5f1d6fdd3b23aeb11a236fb3f45a WatchSource:0}: Error finding container e8f9177ac258f692a581f1d6422ac5e6be0a5f1d6fdd3b23aeb11a236fb3f45a: Status 404 returned error can't find the container with id e8f9177ac258f692a581f1d6422ac5e6be0a5f1d6fdd3b23aeb11a236fb3f45a
Oct 29 22:52:32.671408 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.671386    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"]
Oct 29 22:52:32.673934 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:32.673907    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bcb0b6_9f48_4a6e_bd40_111de6a3cf23.slice/crio-3a124bd43397f17dab22de4648d27af4613adacf36251ff9fe57aafba90ea6dc WatchSource:0}: Error finding container 3a124bd43397f17dab22de4648d27af4613adacf36251ff9fe57aafba90ea6dc: Status 404 returned error can't find the container with id 3a124bd43397f17dab22de4648d27af4613adacf36251ff9fe57aafba90ea6dc
Oct 29 22:52:32.905659 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.905631    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"]
Oct 29 22:52:32.909003 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.908982    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:32.928340 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.928318    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-5bf55a\""
Oct 29 22:52:32.932735 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.932714    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"]
Oct 29 22:52:32.935326 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:32.935309    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-5bf55a-dockercfg-wztfx\""
Oct 29 22:52:33.058068 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.058035    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbns\" (UniqueName: \"kubernetes.io/projected/fb308604-255e-4f3e-8c89-8b9b571e19da-kube-api-access-clbns\") pod \"payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm\" (UID: \"fb308604-255e-4f3e-8c89-8b9b571e19da\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:33.058230 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.058097    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb308604-255e-4f3e-8c89-8b9b571e19da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm\" (UID: \"fb308604-255e-4f3e-8c89-8b9b571e19da\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:33.158708 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.158626    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbns\" (UniqueName: \"kubernetes.io/projected/fb308604-255e-4f3e-8c89-8b9b571e19da-kube-api-access-clbns\") pod \"payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm\" (UID: \"fb308604-255e-4f3e-8c89-8b9b571e19da\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:33.158708 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.158678    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb308604-255e-4f3e-8c89-8b9b571e19da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm\" (UID: \"fb308604-255e-4f3e-8c89-8b9b571e19da\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:33.159345 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.159327    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb308604-255e-4f3e-8c89-8b9b571e19da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm\" (UID: \"fb308604-255e-4f3e-8c89-8b9b571e19da\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:33.178336 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.178297    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr" event={"ID":"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23","Type":"ContainerStarted","Data":"a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa"}
Oct 29 22:52:33.178727 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.178372    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr" event={"ID":"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23","Type":"ContainerStarted","Data":"3a124bd43397f17dab22de4648d27af4613adacf36251ff9fe57aafba90ea6dc"}
Oct 29 22:52:33.178727 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.178404    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:33.178838 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.178811    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbns\" (UniqueName: \"kubernetes.io/projected/fb308604-255e-4f3e-8c89-8b9b571e19da-kube-api-access-clbns\") pod \"payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm\" (UID: \"fb308604-255e-4f3e-8c89-8b9b571e19da\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:33.179871 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.179850    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4" event={"ID":"f952eacf-1713-4dba-890b-ac9838b1636b","Type":"ContainerStarted","Data":"7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24"}
Oct 29 22:52:33.179964 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.179890    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4" event={"ID":"f952eacf-1713-4dba-890b-ac9838b1636b","Type":"ContainerStarted","Data":"e8f9177ac258f692a581f1d6422ac5e6be0a5f1d6fdd3b23aeb11a236fb3f45a"}
Oct 29 22:52:33.179964 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.179910    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:33.220807 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.220751    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr" podStartSLOduration=2.220736661 podStartE2EDuration="2.220736661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:33.219917803 +0000 UTC m=+610.360659410" watchObservedRunningTime="2026-10-29 22:52:33.220736661 +0000 UTC m=+610.361478270"
Oct 29 22:52:33.221026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.221004    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:33.272987 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.272149    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4" podStartSLOduration=2.27213022 podStartE2EDuration="2.27213022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:33.27152673 +0000 UTC m=+610.412268315" watchObservedRunningTime="2026-10-29 22:52:33.27213022 +0000 UTC m=+610.412871805"
Oct 29 22:52:33.513796 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:33.513773    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"]
Oct 29 22:52:33.515911 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:33.515877    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb308604_255e_4f3e_8c89_8b9b571e19da.slice/crio-4bba2981cdf31438b800924d3e13952d52dab0285cc3f9e7337d574c2d4d106a WatchSource:0}: Error finding container 4bba2981cdf31438b800924d3e13952d52dab0285cc3f9e7337d574c2d4d106a: Status 404 returned error can't find the container with id 4bba2981cdf31438b800924d3e13952d52dab0285cc3f9e7337d574c2d4d106a
Oct 29 22:52:34.190371 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:34.190330    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm" event={"ID":"fb308604-255e-4f3e-8c89-8b9b571e19da","Type":"ContainerStarted","Data":"f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf"}
Oct 29 22:52:34.190795 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:34.190399    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm" event={"ID":"fb308604-255e-4f3e-8c89-8b9b571e19da","Type":"ContainerStarted","Data":"4bba2981cdf31438b800924d3e13952d52dab0285cc3f9e7337d574c2d4d106a"}
Oct 29 22:52:34.190795 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:34.190530    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:34.231360 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:34.231297    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm" podStartSLOduration=2.231279797 podStartE2EDuration="2.231279797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:34.23083878 +0000 UTC m=+611.371580389" watchObservedRunningTime="2026-10-29 22:52:34.231279797 +0000 UTC m=+611.372021381"
Oct 29 22:52:37.044626 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.021552    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"]
Oct 29 22:52:37.045329 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.045304    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:37.053853 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.053823    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"]
Oct 29 22:52:37.084251 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.084219    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-81f92d\""
Oct 29 22:52:37.084389 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.084262    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-81f92d-dockercfg-74697\""
Oct 29 22:52:37.197047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.197010    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pv9v\" (UniqueName: \"kubernetes.io/projected/2e961f1d-5b31-4639-98a8-018c1060b806-kube-api-access-2pv9v\") pod \"payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8\" (UID: \"2e961f1d-5b31-4639-98a8-018c1060b806\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:37.197220 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.197146    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e961f1d-5b31-4639-98a8-018c1060b806-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8\" (UID: \"2e961f1d-5b31-4639-98a8-018c1060b806\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:37.298605 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.298521    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pv9v\" (UniqueName: \"kubernetes.io/projected/2e961f1d-5b31-4639-98a8-018c1060b806-kube-api-access-2pv9v\") pod \"payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8\" (UID: \"2e961f1d-5b31-4639-98a8-018c1060b806\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:37.298605 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.298576    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e961f1d-5b31-4639-98a8-018c1060b806-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8\" (UID: \"2e961f1d-5b31-4639-98a8-018c1060b806\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:37.299231 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.299213    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e961f1d-5b31-4639-98a8-018c1060b806-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8\" (UID: \"2e961f1d-5b31-4639-98a8-018c1060b806\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:37.318068 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.318042    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pv9v\" (UniqueName: \"kubernetes.io/projected/2e961f1d-5b31-4639-98a8-018c1060b806-kube-api-access-2pv9v\") pod \"payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8\" (UID: \"2e961f1d-5b31-4639-98a8-018c1060b806\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:37.371182 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.371142    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:37.669776 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.669755    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"]
Oct 29 22:52:37.671994 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:37.671971    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e961f1d_5b31_4639_98a8_018c1060b806.slice/crio-24c787757d65b51776188b8b85f6edefbc514cb157649143618dfb122f2ebf35 WatchSource:0}: Error finding container 24c787757d65b51776188b8b85f6edefbc514cb157649143618dfb122f2ebf35: Status 404 returned error can't find the container with id 24c787757d65b51776188b8b85f6edefbc514cb157649143618dfb122f2ebf35
Oct 29 22:52:37.879233 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.879198    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"]
Oct 29 22:52:37.882433 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.882415    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:37.900967 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.900941    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-dd3ff3-dockercfg-5wkmv\""
Oct 29 22:52:37.903301 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.903276    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"]
Oct 29 22:52:37.935026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.934957    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-dd3ff3\""
Oct 29 22:52:37.966996 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.966966    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"]
Oct 29 22:52:37.970478 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.970462    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:37.996144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:37.990980    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"]
Oct 29 22:52:38.005226 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.005191    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5620b0a7-c9a8-4652-b202-d522ac93f01b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798\" (UID: \"5620b0a7-c9a8-4652-b202-d522ac93f01b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:38.005382 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.005352    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796sf\" (UniqueName: \"kubernetes.io/projected/5620b0a7-c9a8-4652-b202-d522ac93f01b-kube-api-access-796sf\") pod \"payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798\" (UID: \"5620b0a7-c9a8-4652-b202-d522ac93f01b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:38.106441 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.106409    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5620b0a7-c9a8-4652-b202-d522ac93f01b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798\" (UID: \"5620b0a7-c9a8-4652-b202-d522ac93f01b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:38.106932 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.106486    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796sf\" (UniqueName: \"kubernetes.io/projected/5620b0a7-c9a8-4652-b202-d522ac93f01b-kube-api-access-796sf\") pod \"payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798\" (UID: \"5620b0a7-c9a8-4652-b202-d522ac93f01b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:38.106932 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.106580    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbvn\" (UniqueName: \"kubernetes.io/projected/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-kube-api-access-7lbvn\") pod \"payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf\" (UID: \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:38.106932 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.106654    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf\" (UID: \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:38.107297 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.107274    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5620b0a7-c9a8-4652-b202-d522ac93f01b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798\" (UID: \"5620b0a7-c9a8-4652-b202-d522ac93f01b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:38.125724 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.125704    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-796sf\" (UniqueName: \"kubernetes.io/projected/5620b0a7-c9a8-4652-b202-d522ac93f01b-kube-api-access-796sf\") pod \"payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798\" (UID: \"5620b0a7-c9a8-4652-b202-d522ac93f01b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:38.195328 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.195249    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:38.207586 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.207557    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbvn\" (UniqueName: \"kubernetes.io/projected/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-kube-api-access-7lbvn\") pod \"payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf\" (UID: \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:38.207697 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.207598    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf\" (UID: \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:38.208224 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.208206    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf\" (UID: \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:38.215483 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.215457    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8" event={"ID":"2e961f1d-5b31-4639-98a8-018c1060b806","Type":"ContainerStarted","Data":"bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb"}
Oct 29 22:52:38.215580 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.215510    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8" event={"ID":"2e961f1d-5b31-4639-98a8-018c1060b806","Type":"ContainerStarted","Data":"24c787757d65b51776188b8b85f6edefbc514cb157649143618dfb122f2ebf35"}
Oct 29 22:52:38.215580 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.215558    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:38.226345 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.226324    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbvn\" (UniqueName: \"kubernetes.io/projected/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-kube-api-access-7lbvn\") pod \"payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf\" (UID: \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:38.253617 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.253557    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8" podStartSLOduration=2.253538267 podStartE2EDuration="2.253538267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:38.251581083 +0000 UTC m=+615.392322668" watchObservedRunningTime="2026-10-29 22:52:38.253538267 +0000 UTC m=+615.394279852"
Oct 29 22:52:38.283441 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.283409    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:38.489251 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.489224    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"]
Oct 29 22:52:38.491253 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:38.491221    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5620b0a7_c9a8_4652_b202_d522ac93f01b.slice/crio-d2328eb2c29ae613e3323b194b2b297cd6f424303b9691b3d6c40b358900b204 WatchSource:0}: Error finding container d2328eb2c29ae613e3323b194b2b297cd6f424303b9691b3d6c40b358900b204: Status 404 returned error can't find the container with id d2328eb2c29ae613e3323b194b2b297cd6f424303b9691b3d6c40b358900b204
Oct 29 22:52:38.579845 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:38.579819    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"]
Oct 29 22:52:38.581318 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:38.581280    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e6398e_bbf3_4fe6_a5e5_f6e617a89f26.slice/crio-34b27d0ac94a264a4f43bd9b5dac76ff33e0161007559b1f2c1811a99db54354 WatchSource:0}: Error finding container 34b27d0ac94a264a4f43bd9b5dac76ff33e0161007559b1f2c1811a99db54354: Status 404 returned error can't find the container with id 34b27d0ac94a264a4f43bd9b5dac76ff33e0161007559b1f2c1811a99db54354
Oct 29 22:52:39.222205 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:39.222167    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf" event={"ID":"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26","Type":"ContainerStarted","Data":"676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee"}
Oct 29 22:52:39.222666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:39.222222    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf" event={"ID":"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26","Type":"ContainerStarted","Data":"34b27d0ac94a264a4f43bd9b5dac76ff33e0161007559b1f2c1811a99db54354"}
Oct 29 22:52:39.222666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:39.222246    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:39.223743 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:39.223715    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798" event={"ID":"5620b0a7-c9a8-4652-b202-d522ac93f01b","Type":"ContainerStarted","Data":"fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207"}
Oct 29 22:52:39.223854 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:39.223760    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798" event={"ID":"5620b0a7-c9a8-4652-b202-d522ac93f01b","Type":"ContainerStarted","Data":"d2328eb2c29ae613e3323b194b2b297cd6f424303b9691b3d6c40b358900b204"}
Oct 29 22:52:39.223936 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:39.223925    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:39.278197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:39.278152    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf" podStartSLOduration=2.278136219 podStartE2EDuration="2.278136219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:39.277776302 +0000 UTC m=+616.418517888" watchObservedRunningTime="2026-10-29 22:52:39.278136219 +0000 UTC m=+616.418877803"
Oct 29 22:52:39.316178 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:39.316124    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798" podStartSLOduration=2.316110663 podStartE2EDuration="2.316110663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:39.314679663 +0000 UTC m=+616.455421248" watchObservedRunningTime="2026-10-29 22:52:39.316110663 +0000 UTC m=+616.456852244"
Oct 29 22:52:42.175961 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.175928    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 22:52:42.303867 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.303837    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"]
Oct 29 22:52:42.307284 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.307266    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.327671 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.327578    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 22:52:42.328588 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.328561    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-5bf55a-serving-cert\""
Oct 29 22:52:42.329431 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.329410    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:52:42.329858 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.329813    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:52:42.331144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.331122    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"]
Oct 29 22:52:42.432218 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.432143    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7l5lq\""
Oct 29 22:52:42.454767 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.454741    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb920ed-428a-4ddd-8aac-ac2d7f931463-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.454979 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.454776    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-maas-api-tls\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.454979 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.454955    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwxq4\" (UniqueName: \"kubernetes.io/projected/0cb920ed-428a-4ddd-8aac-ac2d7f931463-kube-api-access-rwxq4\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.455124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.454990    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-metrics-certs\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.556032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.555998    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb920ed-428a-4ddd-8aac-ac2d7f931463-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.556239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.556040    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-maas-api-tls\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.556239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.556140    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwxq4\" (UniqueName: \"kubernetes.io/projected/0cb920ed-428a-4ddd-8aac-ac2d7f931463-kube-api-access-rwxq4\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.556239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.556159    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-metrics-certs\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.556750 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.556722    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb920ed-428a-4ddd-8aac-ac2d7f931463-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.558524 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.558495    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-metrics-certs\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.558632 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.558549    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-maas-api-tls\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.576033 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.576010    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwxq4\" (UniqueName: \"kubernetes.io/projected/0cb920ed-428a-4ddd-8aac-ac2d7f931463-kube-api-access-rwxq4\") pod \"maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.620046 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.620019    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:42.917520 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:42.917494    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"]
Oct 29 22:52:42.919989 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:42.919952    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb920ed_428a_4ddd_8aac_ac2d7f931463.slice/crio-a2397ea0721da8b6ed45bc621164e455ca8287b35e14e4cda216ae3736b5319b WatchSource:0}: Error finding container a2397ea0721da8b6ed45bc621164e455ca8287b35e14e4cda216ae3736b5319b: Status 404 returned error can't find the container with id a2397ea0721da8b6ed45bc621164e455ca8287b35e14e4cda216ae3736b5319b
Oct 29 22:52:43.242748 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:43.242654    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk" event={"ID":"0cb920ed-428a-4ddd-8aac-ac2d7f931463","Type":"ContainerStarted","Data":"a2397ea0721da8b6ed45bc621164e455ca8287b35e14e4cda216ae3736b5319b"}
Oct 29 22:52:44.193820 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:44.193790    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 22:52:44.194332 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:44.194295    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 22:52:45.199289 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:45.199264    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 22:52:46.261537 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:46.261503    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk" event={"ID":"0cb920ed-428a-4ddd-8aac-ac2d7f931463","Type":"ContainerStarted","Data":"184b3c90a9ef16d8c05c1327c4b7ba767781e8744c629f59246d3f900efad5cf"}
Oct 29 22:52:46.261966 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:46.261635    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:46.305887 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:46.305830    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk" podStartSLOduration=2.037080066 podStartE2EDuration="4.305813021s" totalImagesPullingTime="2.268732955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:46.304402795 +0000 UTC m=+623.445144383" watchObservedRunningTime="2026-10-29 22:52:46.305813021 +0000 UTC m=+623.446554630"
Oct 29 22:52:49.226906 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:49.226873    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 22:52:50.231698 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:50.231666    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:52:50.232066 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:50.231731    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:52:52.296111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:52.295244    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 22:52:53.122922 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:52:53.122884    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:52:55.896158 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.896119    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"]
Oct 29 22:52:55.900792 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.900772    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:55.918720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.918696    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-20f88f\"/\"openshift-service-ca.crt\""
Oct 29 22:52:55.930814 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.930790    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-20f88f\"/\"default-dockercfg-l88xn\""
Oct 29 22:52:55.930814 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.930790    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-20f88f\"/\"kube-root-ca.crt\""
Oct 29 22:52:55.930814 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.930833    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-20f88f\"/\"simulator-w1-20f88f-kserve-self-signed-certs\""
Oct 29 22:52:55.933787 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.933765    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"]
Oct 29 22:52:55.954758 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.954735    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9sz\" (UniqueName: \"kubernetes.io/projected/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-kube-api-access-gb9sz\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:55.954857 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.954785    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-model-cache\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:55.954913 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.954853    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-dshm\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:55.954913 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.954872    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-home\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:55.955033 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.955008    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tmp-dir\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:55.955080 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:55.955054    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tls-certs\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.055507 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.055473    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-model-cache\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.055507 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.055514    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-dshm\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.055759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.055536    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-home\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.055759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.055589    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tmp-dir\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.056154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.055909    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tls-certs\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.056154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.055965    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-model-cache\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.056154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.055999    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tmp-dir\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.056154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.056076    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-home\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.056154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.056121    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9sz\" (UniqueName: \"kubernetes.io/projected/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-kube-api-access-gb9sz\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.084271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.084146    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-dshm\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.084698 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.084641    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tls-certs\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.087171 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.087137    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9sz\" (UniqueName: \"kubernetes.io/projected/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-kube-api-access-gb9sz\") pod \"simulator-w1-20f88f-kserve-6dbff847f8-zvjvb\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") " pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.215150 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.215058    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:56.598146 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:56.585620    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"]
Oct 29 22:52:56.598146 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:52:56.587185    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fbee0a3_5948_4aea_a1bc_b07f6c2cd54f.slice/crio-81d39d48647b6a1beae187fb7908ce6ab8d85ed3200e3a00b502195cf9f38cca WatchSource:0}: Error finding container 81d39d48647b6a1beae187fb7908ce6ab8d85ed3200e3a00b502195cf9f38cca: Status 404 returned error can't find the container with id 81d39d48647b6a1beae187fb7908ce6ab8d85ed3200e3a00b502195cf9f38cca
Oct 29 22:52:57.150618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.150581    4742 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:52:57.150963 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.150659    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:52:57.355345 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.355314    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" event={"ID":"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f","Type":"ContainerStarted","Data":"b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3"}
Oct 29 22:52:57.356490 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.355359    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" event={"ID":"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f","Type":"ContainerStarted","Data":"81d39d48647b6a1beae187fb7908ce6ab8d85ed3200e3a00b502195cf9f38cca"}
Oct 29 22:52:57.357054 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.357034    4742 generic.go:356] "Generic (PLEG): container finished" podID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerID="db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82" exitCode=0
Oct 29 22:52:57.357188 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.357062    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" event={"ID":"e57f7435-1e6b-4e25-af1a-c399471ca25d","Type":"ContainerDied","Data":"db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82"}
Oct 29 22:52:57.371111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.357470    4742 scope.go:122] "RemoveContainer" containerID="db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82"
Oct 29 22:52:57.383955 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.383917    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:57.384067 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.384019    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:57.384147 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.384067    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:57.405057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:57.405013    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podStartSLOduration=2.404998936 podStartE2EDuration="2.404998936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:57.403451598 +0000 UTC m=+634.544193184" watchObservedRunningTime="2026-10-29 22:52:57.404998936 +0000 UTC m=+634.545740519"
Oct 29 22:52:58.155564 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.155531    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:52:58.155922 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.155579    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:52:58.215749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.215719    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:52:58.384369 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.384336    4742 generic.go:356] "Generic (PLEG): container finished" podID="091431d4-cb1e-46ef-babb-e5f54e589476" containerID="255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40" exitCode=0
Oct 29 22:52:58.384581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.384419    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" event={"ID":"091431d4-cb1e-46ef-babb-e5f54e589476","Type":"ContainerDied","Data":"255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40"}
Oct 29 22:52:58.385283 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.385197    4742 scope.go:122] "RemoveContainer" containerID="255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40"
Oct 29 22:52:58.386581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.386552    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:58.386667 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.386622    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:58.386667 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.386655    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:52:58.387747 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:58.387723    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" event={"ID":"e57f7435-1e6b-4e25-af1a-c399471ca25d","Type":"ContainerStarted","Data":"610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5"}
Oct 29 22:52:59.124184 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:59.124100    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:52:59.124389 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:59.124282    4742 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:52:59.124389 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:59.124334    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:52:59.412163 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:59.411146    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" event={"ID":"091431d4-cb1e-46ef-babb-e5f54e589476","Type":"ContainerStarted","Data":"70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4"}
Oct 29 22:52:59.722158 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:59.721486    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:52:59.722158 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:59.721638    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:52:59.722158 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:52:59.721684    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:53:00.129183 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:00.126478    4742 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:00.129183 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:00.126545    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:53:00.722110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:00.721444    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:53:00.722110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:00.721503    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:53:01.124362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:01.124332    4742 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:01.124529 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:01.124380    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:53:01.722509 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:01.722474    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:53:01.722902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:01.722529    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:53:02.124693 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:02.124234    4742 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:02.124693 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:02.124304    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:53:02.722308 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:02.722269    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:53:02.722493 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:02.722342    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:53:03.124857 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:03.124817    4742 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:03.125360 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:03.124874    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 22:53:03.721789 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:03.721750    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 22:53:03.722099 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:03.721811    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 22:53:04.128636 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.128610    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:53:04.129050 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.128952    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:53:04.129563 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.129548    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 22:53:04.445675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.445584    4742 generic.go:356] "Generic (PLEG): container finished" podID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerID="e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294" exitCode=0
Oct 29 22:53:04.445675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.445657    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" event={"ID":"7c4a636e-a3b0-4347-8b47-68c6b6dca509","Type":"ContainerDied","Data":"e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294"}
Oct 29 22:53:04.446216 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.446195    4742 scope.go:122] "RemoveContainer" containerID="e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294"
Oct 29 22:53:04.447559 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.447535    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:53:04.447617 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.447600    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:53:04.447652 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.447632    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:53:04.873037 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.872897    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:53:04.874266 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.874229    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:53:04.875042 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:04.875020    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 22:53:05.451829 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:05.451794    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" event={"ID":"7c4a636e-a3b0-4347-8b47-68c6b6dca509","Type":"ContainerStarted","Data":"af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02"}
Oct 29 22:53:05.985984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:05.985949    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"]
Oct 29 22:53:05.993753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:05.993725    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"]
Oct 29 22:53:05.993927 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:05.993820    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.009030 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.009009    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"kube-root-ca.crt\""
Oct 29 22:53:06.009262 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.009030    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"openshift-service-ca.crt\""
Oct 29 22:53:06.020725 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.020689    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"default-dockercfg-spmz9\""
Oct 29 22:53:06.020725 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.020702    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"simulator-w4-dd3ff3-kserve-self-signed-certs\""
Oct 29 22:53:06.035486 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.035462    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-model-cache\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.035612 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.035506    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tls-certs\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.035612 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.035526    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-dshm\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.035697 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.035606    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k57l\" (UniqueName: \"kubernetes.io/projected/1dfce1f2-1214-4926-bfeb-5f651d4416d6-kube-api-access-4k57l\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.035697 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.035657    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-home\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.035697 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.035688    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tmp-dir\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.136879 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.136839    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tmp-dir\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.137057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.136910    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-model-cache\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.137057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.136949    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tls-certs\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.137057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.136967    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-dshm\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.137057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.136987    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k57l\" (UniqueName: \"kubernetes.io/projected/1dfce1f2-1214-4926-bfeb-5f651d4416d6-kube-api-access-4k57l\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.137057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.137044    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-home\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.137357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.137335    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tmp-dir\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.137418 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.137377    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-model-cache\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.137481 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.137444    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-home\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.139343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.139318    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-dshm\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.139547 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.139528    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tls-certs\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.155295 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.155268    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k57l\" (UniqueName: \"kubernetes.io/projected/1dfce1f2-1214-4926-bfeb-5f651d4416d6-kube-api-access-4k57l\") pod \"simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") " pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.215963 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.215735    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:53:06.233297 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.233268    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:53:06.313915 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.313813    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:06.418880 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.418541    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:53:06.418880 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.418635    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:53:06.418880 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.418669    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:53:06.492077 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.492047    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:53:06.634792 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:06.634751    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"]
Oct 29 22:53:06.641412 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:53:06.641321    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dfce1f2_1214_4926_bfeb_5f651d4416d6.slice/crio-f1ba3461d0ce114e8ca910e5fc3244f31a442373cbc56dd917b534536f310887 WatchSource:0}: Error finding container f1ba3461d0ce114e8ca910e5fc3244f31a442373cbc56dd917b534536f310887: Status 404 returned error can't find the container with id f1ba3461d0ce114e8ca910e5fc3244f31a442373cbc56dd917b534536f310887
Oct 29 22:53:07.417828 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.417788    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:53:07.418040 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.417850    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:53:07.485646 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.485604    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" event={"ID":"1dfce1f2-1214-4926-bfeb-5f651d4416d6","Type":"ContainerStarted","Data":"ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32"}
Oct 29 22:53:07.485646 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.485651    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" event={"ID":"1dfce1f2-1214-4926-bfeb-5f651d4416d6","Type":"ContainerStarted","Data":"f1ba3461d0ce114e8ca910e5fc3244f31a442373cbc56dd917b534536f310887"}
Oct 29 22:53:07.526138 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.526073    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podStartSLOduration=2.526059377 podStartE2EDuration="2.526059377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:07.525697826 +0000 UTC m=+644.666439411" watchObservedRunningTime="2026-10-29 22:53:07.526059377 +0000 UTC m=+644.666800960"
Oct 29 22:53:07.589051 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.589021    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"]
Oct 29 22:53:07.593141 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.593117    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.609353 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.609327    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"openshift-service-ca.crt\""
Oct 29 22:53:07.609921 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.609898    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"kube-root-ca.crt\""
Oct 29 22:53:07.610065 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.609997    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"default-dockercfg-r9mrg\""
Oct 29 22:53:07.610195 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.610069    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"simulator-w3-07eef2-kserve-self-signed-certs\""
Oct 29 22:53:07.612267 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.612244    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"]
Oct 29 22:53:07.653225 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.653194    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjc7q\" (UniqueName: \"kubernetes.io/projected/278c89c3-74ca-422d-b7c0-e13880664f59-kube-api-access-tjc7q\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.653225 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.653229    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-dshm\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.653477 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.653250    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-model-cache\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.653477 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.653395    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-tmp-dir\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.653477 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.653457    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-home\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.653572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.653488    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278c89c3-74ca-422d-b7c0-e13880664f59-tls-certs\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.754700 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.754621    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-tmp-dir\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.754700 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.754693    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-home\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.754931 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.754711    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278c89c3-74ca-422d-b7c0-e13880664f59-tls-certs\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.754931 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.754742    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjc7q\" (UniqueName: \"kubernetes.io/projected/278c89c3-74ca-422d-b7c0-e13880664f59-kube-api-access-tjc7q\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.754931 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.754761    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-dshm\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.754931 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.754841    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-model-cache\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.755161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.755060    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-tmp-dir\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.755161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.755129    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-home\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.755228 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.755195    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-model-cache\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.756933 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.756910    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-dshm\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.757121 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.757081    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278c89c3-74ca-422d-b7c0-e13880664f59-tls-certs\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.786743 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.786711    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjc7q\" (UniqueName: \"kubernetes.io/projected/278c89c3-74ca-422d-b7c0-e13880664f59-kube-api-access-tjc7q\") pod \"simulator-w3-07eef2-kserve-65576c47f-2gcpn\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") " pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:07.906668 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:07.906636    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:08.218656 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:08.218630    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"]
Oct 29 22:53:08.220719 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:53:08.220689    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod278c89c3_74ca_422d_b7c0_e13880664f59.slice/crio-330ad115bb77d1054083dc25b935fab343c51bc522435f76ffa3750e8ca96c7a WatchSource:0}: Error finding container 330ad115bb77d1054083dc25b935fab343c51bc522435f76ffa3750e8ca96c7a: Status 404 returned error can't find the container with id 330ad115bb77d1054083dc25b935fab343c51bc522435f76ffa3750e8ca96c7a
Oct 29 22:53:08.314619 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:08.314586    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:08.417550 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:08.417512    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:53:08.417726 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:08.417573    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:53:08.492590 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:08.492503    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" event={"ID":"278c89c3-74ca-422d-b7c0-e13880664f59","Type":"ContainerStarted","Data":"977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9"}
Oct 29 22:53:08.492590 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:08.492543    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" event={"ID":"278c89c3-74ca-422d-b7c0-e13880664f59","Type":"ContainerStarted","Data":"330ad115bb77d1054083dc25b935fab343c51bc522435f76ffa3750e8ca96c7a"}
Oct 29 22:53:08.535624 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:08.535557    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podStartSLOduration=1.535535437 podStartE2EDuration="1.535535437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:08.532020986 +0000 UTC m=+645.672762571" watchObservedRunningTime="2026-10-29 22:53:08.535535437 +0000 UTC m=+645.676277023"
Oct 29 22:53:08.914223 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:08.912629    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:09.416886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:09.416853    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:53:09.417058 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:09.416908    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:53:10.417298 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:10.417252    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:53:10.417696 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:10.417324    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 22:53:11.421744 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:11.421711    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:53:11.422229 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:11.422003    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:53:11.422771 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:11.422751    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 22:53:16.317432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:16.317395    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:16.357109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:16.355568    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:16.544916 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:16.544888    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:53:17.906962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:17.906922    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:17.919856 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:17.919831    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:18.554724 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:18.554695    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:53:19.113180 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.113140    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"]
Oct 29 22:53:19.117776 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.117747    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.137853 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.137815    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"]
Oct 29 22:53:19.146122 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.146047    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ffba57\"/\"kube-root-ca.crt\""
Oct 29 22:53:19.146362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.146123    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ffba57\"/\"openshift-service-ca.crt\""
Oct 29 22:53:19.146362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.146123    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ffba57\"/\"simulator-w0-ffba57-kserve-self-signed-certs\""
Oct 29 22:53:19.146510 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.146143    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ffba57\"/\"default-dockercfg-h5mgq\""
Oct 29 22:53:19.249611 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.249570    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgld\" (UniqueName: \"kubernetes.io/projected/920def85-325f-4d23-b508-2b036320e3bb-kube-api-access-fcgld\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.249786 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.249625    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-tmp-dir\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.249786 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.249752    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-dshm\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.249902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.249799    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-home\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.249902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.249815    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920def85-325f-4d23-b508-2b036320e3bb-tls-certs\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.250039 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.249936    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-model-cache\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.365984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.365127    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgld\" (UniqueName: \"kubernetes.io/projected/920def85-325f-4d23-b508-2b036320e3bb-kube-api-access-fcgld\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.365984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.365186    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-tmp-dir\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.365984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.365259    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-dshm\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.365984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.365305    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-home\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.365984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.365330    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920def85-325f-4d23-b508-2b036320e3bb-tls-certs\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.365984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.365383    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-model-cache\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.365984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.365836    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-model-cache\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.367203 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.367177    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-home\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.367743 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.367704    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-tmp-dir\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.370904 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.370497    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920def85-325f-4d23-b508-2b036320e3bb-tls-certs\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.373277 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.373250    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-dshm\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.385254 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.385221    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgld\" (UniqueName: \"kubernetes.io/projected/920def85-325f-4d23-b508-2b036320e3bb-kube-api-access-fcgld\") pod \"simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") " pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.430991 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.430959    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:19.737653 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:19.737624    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"]
Oct 29 22:53:19.739472 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:53:19.739441    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod920def85_325f_4d23_b508_2b036320e3bb.slice/crio-bee3cdbc518d0760a24c5a931b4845c56b7e768d45a9377913c1b797acfa40a6 WatchSource:0}: Error finding container bee3cdbc518d0760a24c5a931b4845c56b7e768d45a9377913c1b797acfa40a6: Status 404 returned error can't find the container with id bee3cdbc518d0760a24c5a931b4845c56b7e768d45a9377913c1b797acfa40a6
Oct 29 22:53:20.555157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:20.555116    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" event={"ID":"920def85-325f-4d23-b508-2b036320e3bb","Type":"ContainerStarted","Data":"aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be"}
Oct 29 22:53:20.555554 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:20.555170    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" event={"ID":"920def85-325f-4d23-b508-2b036320e3bb","Type":"ContainerStarted","Data":"bee3cdbc518d0760a24c5a931b4845c56b7e768d45a9377913c1b797acfa40a6"}
Oct 29 22:53:20.597986 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:20.595313    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podStartSLOduration=1.5952937550000001 podStartE2EDuration="1.595293755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:20.594121294 +0000 UTC m=+657.734862874" watchObservedRunningTime="2026-10-29 22:53:20.595293755 +0000 UTC m=+657.736035339"
Oct 29 22:53:21.431718 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:21.431688    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:23.122204 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:53:23.122168    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:53:29.431560 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:29.431524    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:29.444581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:29.444555    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:29.609081 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:29.609051    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:53:39.338946 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.338854    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"]
Oct 29 22:53:39.343381 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.343350    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.370592 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.370563    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"premium-w4-dd3ff3-kserve-self-signed-certs\""
Oct 29 22:53:39.372543 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.372515    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"]
Oct 29 22:53:39.468315 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.468280    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-home\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.468315 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.468325    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tmp-dir\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.468583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.468473    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-dshm\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.468583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.468518    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tls-certs\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.468583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.468553    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-model-cache\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.468756 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.468668    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8z9\" (UniqueName: \"kubernetes.io/projected/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-kube-api-access-8n8z9\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.569620 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.569586    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-home\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.569818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.569634    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tmp-dir\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.569818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.569698    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-dshm\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.569818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.569718    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tls-certs\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.569818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.569737    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-model-cache\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.569818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.569758    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8z9\" (UniqueName: \"kubernetes.io/projected/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-kube-api-access-8n8z9\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.570207 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.570179    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tmp-dir\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.570285 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.570224    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-home\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.570491 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.570460    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-model-cache\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.572581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.572557    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-dshm\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.573257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.573237    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tls-certs\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.595418 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.595361    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8z9\" (UniqueName: \"kubernetes.io/projected/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-kube-api-access-8n8z9\") pod \"premium-w4-dd3ff3-kserve-f7fb76964-5lswf\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") " pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.656233 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.656211    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:39.950236 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:39.950214    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"]
Oct 29 22:53:39.952753 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:53:39.952726    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e5151b2_2b0c_4feb_9069_ac0cbbba3dc2.slice/crio-5d085f7828aa1e054c13bbe9a23bfd48d895fe8e919fe4894bbe274cab3b47a0 WatchSource:0}: Error finding container 5d085f7828aa1e054c13bbe9a23bfd48d895fe8e919fe4894bbe274cab3b47a0: Status 404 returned error can't find the container with id 5d085f7828aa1e054c13bbe9a23bfd48d895fe8e919fe4894bbe274cab3b47a0
Oct 29 22:53:40.652493 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:40.652463    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" event={"ID":"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2","Type":"ContainerStarted","Data":"fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67"}
Oct 29 22:53:40.652879 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:40.652502    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" event={"ID":"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2","Type":"ContainerStarted","Data":"5d085f7828aa1e054c13bbe9a23bfd48d895fe8e919fe4894bbe274cab3b47a0"}
Oct 29 22:53:40.656785 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:40.656760    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:40.697856 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:40.697804    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podStartSLOduration=1.6977908309999998 podStartE2EDuration="1.697790831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:40.69592382 +0000 UTC m=+677.836665404" watchObservedRunningTime="2026-10-29 22:53:40.697790831 +0000 UTC m=+677.838532436"
Oct 29 22:53:43.217121 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.217063    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"]
Oct 29 22:53:43.221350 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.221322    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.248271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.248241    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"openshift-service-ca.crt\""
Oct 29 22:53:43.248474 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.248323    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"kube-root-ca.crt\""
Oct 29 22:53:43.248474 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.248323    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"premium-w6-81f92d-kserve-self-signed-certs\""
Oct 29 22:53:43.248606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.248404    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"default-dockercfg-nmcsr\""
Oct 29 22:53:43.250628 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.250606    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"]
Oct 29 22:53:43.306767 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.306737    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-dshm\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.306913 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.306795    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-home\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.306952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.306914    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-model-cache\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.306993 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.306965    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tmp-dir\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.307108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.307071    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2bt\" (UniqueName: \"kubernetes.io/projected/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-kube-api-access-fj2bt\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.307179 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.307151    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tls-certs\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.407823 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.407784    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-home\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.408196 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.408157    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-model-cache\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.408359 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.408248    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tmp-dir\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.408359 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.408314    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2bt\" (UniqueName: \"kubernetes.io/projected/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-kube-api-access-fj2bt\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.408459 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.408382    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tls-certs\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.408536 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.408517    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-dshm\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.408691 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.408608    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-home\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.409165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.408903    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tmp-dir\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.411622 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.411577    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-model-cache\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.430124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.430028    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-dshm\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.431657 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.431629    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tls-certs\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.438178 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.438138    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2bt\" (UniqueName: \"kubernetes.io/projected/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-kube-api-access-fj2bt\") pod \"premium-w6-81f92d-kserve-96c68c86d-9bklk\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") " pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.535825 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.535735    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:43.843465 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:43.843435    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"]
Oct 29 22:53:43.850388 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:53:43.850350    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0a9b7f5_7dd9_486f_9d77_c47c4f980957.slice/crio-61f8d5be1f09cd3c14f06504d736e6a8df48b8f5abed68ec7340a61e2ed80937 WatchSource:0}: Error finding container 61f8d5be1f09cd3c14f06504d736e6a8df48b8f5abed68ec7340a61e2ed80937: Status 404 returned error can't find the container with id 61f8d5be1f09cd3c14f06504d736e6a8df48b8f5abed68ec7340a61e2ed80937
Oct 29 22:53:44.674298 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:44.674264    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" event={"ID":"d0a9b7f5-7dd9-486f-9d77-c47c4f980957","Type":"ContainerStarted","Data":"90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774"}
Oct 29 22:53:44.674298 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:44.674304    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" event={"ID":"d0a9b7f5-7dd9-486f-9d77-c47c4f980957","Type":"ContainerStarted","Data":"61f8d5be1f09cd3c14f06504d736e6a8df48b8f5abed68ec7340a61e2ed80937"}
Oct 29 22:53:44.727593 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:44.727543    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podStartSLOduration=1.727530345 podStartE2EDuration="1.727530345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:44.726334921 +0000 UTC m=+681.867076518" watchObservedRunningTime="2026-10-29 22:53:44.727530345 +0000 UTC m=+681.868271929"
Oct 29 22:53:45.536261 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:45.536231    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:49.657279 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:49.657244    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:49.669759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:49.669736    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:49.710680 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:49.710656    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:53:53.131746 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:53:53.131701    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:53:53.536815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:53.536722    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:53.553930 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:53.553896    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:53.730863 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:53.730839    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:53:56.004937 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.004897    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"]
Oct 29 22:53:56.009065 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.009043    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.039323 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.039288    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"default-dockercfg-2mj5c\""
Oct 29 22:53:56.039323 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.039300    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"kube-root-ca.crt\""
Oct 29 22:53:56.039323 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.039302    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"openshift-service-ca.crt\""
Oct 29 22:53:56.039587 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.039301    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"premium-w2-a2e7c8-kserve-self-signed-certs\""
Oct 29 22:53:56.041700 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.041673    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"]
Oct 29 22:53:56.051175 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.051153    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tmp-dir\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.051292 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.051191    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-home\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.051292 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.051217    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttqm\" (UniqueName: \"kubernetes.io/projected/763c2875-5e58-47d8-9b2b-a9b50b8384e0-kube-api-access-7ttqm\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.051370 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.051348    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-dshm\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.051442 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.051424    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tls-certs\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.051589 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.051566    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-model-cache\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.152358 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152324    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-model-cache\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.152544 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152392    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tmp-dir\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.152544 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152418    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-home\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.152544 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152445    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttqm\" (UniqueName: \"kubernetes.io/projected/763c2875-5e58-47d8-9b2b-a9b50b8384e0-kube-api-access-7ttqm\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.152544 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152482    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-dshm\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.152544 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152513    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tls-certs\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.152878 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152842    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tmp-dir\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.153002 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152890    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-model-cache\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.153002 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.152902    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-home\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.154811 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.154790    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-dshm\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.155210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.155189    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tls-certs\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.172786 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.172757    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttqm\" (UniqueName: \"kubernetes.io/projected/763c2875-5e58-47d8-9b2b-a9b50b8384e0-kube-api-access-7ttqm\") pod \"premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") " pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.324606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.324518    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:53:56.623943 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.623915    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"]
Oct 29 22:53:56.625961 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:53:56.625934    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod763c2875_5e58_47d8_9b2b_a9b50b8384e0.slice/crio-1dee8501a8a799f7d7645bd152c98f2d44225503280e0e904ea96c927482af00 WatchSource:0}: Error finding container 1dee8501a8a799f7d7645bd152c98f2d44225503280e0e904ea96c927482af00: Status 404 returned error can't find the container with id 1dee8501a8a799f7d7645bd152c98f2d44225503280e0e904ea96c927482af00
Oct 29 22:53:56.738506 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.738468    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" event={"ID":"763c2875-5e58-47d8-9b2b-a9b50b8384e0","Type":"ContainerStarted","Data":"922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df"}
Oct 29 22:53:56.738686 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.738517    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" event={"ID":"763c2875-5e58-47d8-9b2b-a9b50b8384e0","Type":"ContainerStarted","Data":"1dee8501a8a799f7d7645bd152c98f2d44225503280e0e904ea96c927482af00"}
Oct 29 22:53:56.780353 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:56.780309    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podStartSLOduration=1.78029336 podStartE2EDuration="1.78029336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:56.777865392 +0000 UTC m=+693.918606976" watchObservedRunningTime="2026-10-29 22:53:56.78029336 +0000 UTC m=+693.921034989"
Oct 29 22:53:57.325399 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:53:57.325370    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:54:06.327414 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:06.327374    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:54:06.366996 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:06.366967    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:54:06.804977 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:06.804950    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:54:10.002071 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.002032    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"]
Oct 29 22:54:10.005919 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.005896    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.022205 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.022182    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-20f88f-openshift-default-dockercfg-7q76j\""
Oct 29 22:54:10.036599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.036567    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"]
Oct 29 22:54:10.096130 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096071    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gqf\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-kube-api-access-l6gqf\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.096304 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096171    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7fd15e-1f5c-407a-b86a-677851e1731b-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.096304 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096205    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.096455 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096352    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.096455 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096409    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.096549 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096468    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.096549 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096517    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.096621 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096556    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.096665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.096615    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.197777 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.197740    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7fd15e-1f5c-407a-b86a-677851e1731b-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.197777 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.197783    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.197836    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.197876    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.197902    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.197931    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.197979    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.198003    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198356 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.198038    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gqf\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-kube-api-access-l6gqf\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198356 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.198288    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198445 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.198357    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198477 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.198448    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198556 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.198538    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7fd15e-1f5c-407a-b86a-677851e1731b-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.198673 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.198651    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.200367 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.200340    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.200537 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.200521    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.229077 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.229045    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gqf\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-kube-api-access-l6gqf\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.229077 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.229093    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.319270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.319182    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:10.633392 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.633368    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"]
Oct 29 22:54:10.635068 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:54:10.635041    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7fd15e_1f5c_407a_b86a_677851e1731b.slice/crio-67436276cf670fdf8fa9efdf22835126b725b8b1221455e721bbfe679729a639 WatchSource:0}: Error finding container 67436276cf670fdf8fa9efdf22835126b725b8b1221455e721bbfe679729a639: Status 404 returned error can't find the container with id 67436276cf670fdf8fa9efdf22835126b725b8b1221455e721bbfe679729a639
Oct 29 22:54:10.637455 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.637425    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:10.637530 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.637485    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:10.637530 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.637513    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:10.809910 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.809878    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" event={"ID":"dc7fd15e-1f5c-407a-b86a-677851e1731b","Type":"ContainerStarted","Data":"224fae60d4c495b73f7101b3d7443fe92d31b7b37a21582e5c6c76ee88a06be8"}
Oct 29 22:54:10.810109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.809927    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" event={"ID":"dc7fd15e-1f5c-407a-b86a-677851e1731b","Type":"ContainerStarted","Data":"67436276cf670fdf8fa9efdf22835126b725b8b1221455e721bbfe679729a639"}
Oct 29 22:54:10.855624 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.855561    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podStartSLOduration=1.855541208 podStartE2EDuration="1.855541208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:10.851738702 +0000 UTC m=+707.992480291" watchObservedRunningTime="2026-10-29 22:54:10.855541208 +0000 UTC m=+707.996282793"
Oct 29 22:54:10.897644 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.897559    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"]
Oct 29 22:54:10.901800 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.901781    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:10.919218 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.919191    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-dd3ff3-openshift-default-dockercfg-528h2\""
Oct 29 22:54:10.921968 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:10.921942    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"]
Oct 29 22:54:11.007293 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007259    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvzg\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-kube-api-access-8wvzg\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.007774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007300    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a71d461-c056-47b8-ae80-a12994784a4e-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.007774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007389    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a71d461-c056-47b8-ae80-a12994784a4e-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.007774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007437    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.007774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007469    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.007774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007493    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.007774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007540    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.007774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007630    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.007774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.007668    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.108472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.108428    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a71d461-c056-47b8-ae80-a12994784a4e-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.108472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.108477    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a71d461-c056-47b8-ae80-a12994784a4e-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.108742 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.108505    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.108742 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.108545    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.108742 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.108692    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.108925 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.108754    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.109012 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.108987    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.109127 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.109071    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.109206 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.108998    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.109272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.109225    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.109333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.109283    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.109389 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.109344    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvzg\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-kube-api-access-8wvzg\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.109493 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.109470    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.110019 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.109997    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a71d461-c056-47b8-ae80-a12994784a4e-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.110882 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.110858    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.111004 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.110987    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a71d461-c056-47b8-ae80-a12994784a4e-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.130754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.130718    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvzg\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-kube-api-access-8wvzg\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.131461 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.131430    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.216800 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.216716    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:11.319976 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.319943    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:11.321682 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.321363    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:54:11.321682 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.321414    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 22:54:11.548426 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.548344    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"]
Oct 29 22:54:11.552013 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:54:11.551984    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a71d461_c056_47b8_ae80_a12994784a4e.slice/crio-2f302184b5f74ecb8c992d6091184a06c20c6310147854e96e846623e91293c5 WatchSource:0}: Error finding container 2f302184b5f74ecb8c992d6091184a06c20c6310147854e96e846623e91293c5: Status 404 returned error can't find the container with id 2f302184b5f74ecb8c992d6091184a06c20c6310147854e96e846623e91293c5
Oct 29 22:54:11.554824 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.554796    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:11.554902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.554856    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:11.554902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.554882    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:11.818140 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.818032    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" event={"ID":"6a71d461-c056-47b8-ae80-a12994784a4e","Type":"ContainerStarted","Data":"338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa"}
Oct 29 22:54:11.818140 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:11.818120    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" event={"ID":"6a71d461-c056-47b8-ae80-a12994784a4e","Type":"ContainerStarted","Data":"2f302184b5f74ecb8c992d6091184a06c20c6310147854e96e846623e91293c5"}
Oct 29 22:54:12.217698 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:12.217659    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:12.219298 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:12.219270    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:54:12.219434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:12.219321    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:54:12.320365 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:12.320328    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:54:12.320530 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:12.320390    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 22:54:13.217983 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:13.217934    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:54:13.218494 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:13.218001    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:54:13.320037 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:13.319996    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:54:13.320211 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:13.320055    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 22:54:14.217874 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:14.217840    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:54:14.218045 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:14.217894    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:54:14.320956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:14.320494    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:54:14.320956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:14.320559    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 22:54:15.217884 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:15.217845    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:54:15.218046 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:15.217900    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:54:15.320002 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:15.319965    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:54:15.320426 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:15.320017    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 22:54:16.218133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:16.217633    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:54:16.218133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:16.217717    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:54:16.327330 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:16.327294    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:54:16.327915 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:16.327887    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 22:54:17.217736 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:17.217696    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:54:17.217874 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:17.217767    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:54:17.320134 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:17.320096    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 22:54:17.320324 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:17.320160    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 22:54:18.217768 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:18.217721    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:54:18.218320 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:18.217794    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 22:54:18.537186 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:18.537077    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:18.538783 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:18.538761    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:18.539893 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:18.539872    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 22:54:18.601518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:18.601465    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podStartSLOduration=8.60145346 podStartE2EDuration="8.60145346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:11.87445443 +0000 UTC m=+709.015196028" watchObservedRunningTime="2026-10-29 22:54:18.60145346 +0000 UTC m=+715.742195047"
Oct 29 22:54:20.177944 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:20.177124    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:20.177944 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:20.177820    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:20.179328 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:20.179303    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:54:23.127304 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:54:23.127269    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:54:24.566016 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.565984    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"]
Oct 29 22:54:24.570930 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.570912    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.588530 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.588507    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"]
Oct 29 22:54:24.597642 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.597618    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:24.597748 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.597669    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:54:24.627554 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.627522    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bc7\" (UniqueName: \"kubernetes.io/projected/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-kube-api-access-97bc7\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.627678 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.627572    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.627678 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.627665    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.728521 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.728492    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bc7\" (UniqueName: \"kubernetes.io/projected/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-kube-api-access-97bc7\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.728701 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.728550    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.728701 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.728582    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.729421 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.729400    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.730912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.730890    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.747613 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.747585    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bc7\" (UniqueName: \"kubernetes.io/projected/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-kube-api-access-97bc7\") pod \"maas-api-revoke-sub-38f50282-wxjpm\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:24.883629 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:24.883598    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:25.138753 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.138674    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"]
Oct 29 22:54:25.143337 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.143316    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.157722 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.157703    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-81f92d-openshift-default-dockercfg-x8lwz\""
Oct 29 22:54:25.171724 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.171699    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"]
Oct 29 22:54:25.178067 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.178032    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"]
Oct 29 22:54:25.234583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.234557    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.234707 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.234606    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.234707 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.234698    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.234781 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.234718    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.234781 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.234740    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.234928 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.234907    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.234963 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.234948    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.235029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.235015    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.235063 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.235040    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cds7\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-kube-api-access-2cds7\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336094 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336063    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336246 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336228    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336291 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336270    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cds7\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-kube-api-access-2cds7\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336325 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336299    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336358 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336338    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336419 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336401    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336453 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336436    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336499 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336459    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336499 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336471    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336573 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336549    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336637 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336620    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336678 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336659    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.336828 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.336796    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.337197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.337178    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.338639 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.338619    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.338792 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.338776    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.354876 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.354853    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.354950 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.354928    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cds7\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-kube-api-access-2cds7\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.456291 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.456218    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:25.767399 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.767372    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"]
Oct 29 22:54:25.769600 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:54:25.769577    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc9dad8_a255_4d7a_b627_b4ac73b8b615.slice/crio-931294f10520f6adfda3beceea02f95876d6c4f8091c5450f03c9c6c699e361f WatchSource:0}: Error finding container 931294f10520f6adfda3beceea02f95876d6c4f8091c5450f03c9c6c699e361f: Status 404 returned error can't find the container with id 931294f10520f6adfda3beceea02f95876d6c4f8091c5450f03c9c6c699e361f
Oct 29 22:54:25.772502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.772471    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:25.772579 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.772546    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:25.772579 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.772574    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:25.954399 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.954364    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" event={"ID":"bcc9dad8-a255-4d7a-b627-b4ac73b8b615","Type":"ContainerStarted","Data":"9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43"}
Oct 29 22:54:25.954564 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.954422    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" event={"ID":"bcc9dad8-a255-4d7a-b627-b4ac73b8b615","Type":"ContainerStarted","Data":"931294f10520f6adfda3beceea02f95876d6c4f8091c5450f03c9c6c699e361f"}
Oct 29 22:54:25.955594 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:25.955571    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm" event={"ID":"b603f0e6-fc00-410d-9a9b-bd186e3ff01c","Type":"ContainerStarted","Data":"34d107d575074dab5e5418a5aa8217995383b4b080b50288081b9924652f451f"}
Oct 29 22:54:26.003642 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.003584    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podStartSLOduration=1.003567036 podStartE2EDuration="1.003567036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:25.998279466 +0000 UTC m=+723.139021051" watchObservedRunningTime="2026-10-29 22:54:26.003567036 +0000 UTC m=+723.144308623"
Oct 29 22:54:26.345484 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.345450    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"]
Oct 29 22:54:26.349754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.349727    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.395769 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.395737    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"]
Oct 29 22:54:26.451923 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.451887    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.452106 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.451931    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4xq\" (UniqueName: \"kubernetes.io/projected/c26fda6b-df94-40cd-9b24-d6aad8d90390-kube-api-access-hm4xq\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.452181 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.452108    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.456563 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.456541    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:26.485267 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.485232    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:54:26.485459 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.485297    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:54:26.556983 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.553431    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.556983 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.553481    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4xq\" (UniqueName: \"kubernetes.io/projected/c26fda6b-df94-40cd-9b24-d6aad8d90390-kube-api-access-hm4xq\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.556983 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.553572    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.556983 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.554242    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.556983 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.556918    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.590558 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.590528    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4xq\" (UniqueName: \"kubernetes.io/projected/c26fda6b-df94-40cd-9b24-d6aad8d90390-kube-api-access-hm4xq\") pod \"maas-api-revoke-sub-c373e5a5-q7p9q\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:26.704909 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:26.703604    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:27.028637 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:27.028605    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"]
Oct 29 22:54:27.456745 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:27.456710    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:54:27.456949 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:27.456772    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:54:28.007485 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:28.007446    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q" event={"ID":"c26fda6b-df94-40cd-9b24-d6aad8d90390","Type":"ContainerStarted","Data":"5864bdc215a5cf7e99b44d3922118946b189a7b506d31f3cdb84f615f6ba3b72"}
Oct 29 22:54:28.456658 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:28.456559    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:54:28.456658 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:28.456621    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:54:29.458111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:29.457911    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:54:29.458111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:29.457968    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:54:30.024653 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:30.024608    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q" event={"ID":"c26fda6b-df94-40cd-9b24-d6aad8d90390","Type":"ContainerStarted","Data":"5433c076f8accb9ae0c7b1e70e20c4668a532e7e5b6926de9790af6eb1c8d880"}
Oct 29 22:54:30.026729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:30.026647    4742 generic.go:356] "Generic (PLEG): container finished" podID="b603f0e6-fc00-410d-9a9b-bd186e3ff01c" containerID="0e136ea87b20413cffebdd99603a6cd30bafd7b13da0dd327aaab4203ee8ec11" exitCode=0
Oct 29 22:54:30.026859 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:30.026734    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm" event={"ID":"b603f0e6-fc00-410d-9a9b-bd186e3ff01c","Type":"ContainerDied","Data":"0e136ea87b20413cffebdd99603a6cd30bafd7b13da0dd327aaab4203ee8ec11"}
Oct 29 22:54:30.091112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:30.090295    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q" podStartSLOduration=2.2196754419999998 podStartE2EDuration="4.090272273s" totalImagesPullingTime="1.870596831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:30.084331371 +0000 UTC m=+727.225072957" watchObservedRunningTime="2026-10-29 22:54:30.090272273 +0000 UTC m=+727.231013862"
Oct 29 22:54:30.467873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:30.465546    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:54:30.467873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:30.465613    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:54:31.070103 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.070010    4742 generic.go:356] "Generic (PLEG): container finished" podID="c26fda6b-df94-40cd-9b24-d6aad8d90390" containerID="5433c076f8accb9ae0c7b1e70e20c4668a532e7e5b6926de9790af6eb1c8d880" exitCode=0
Oct 29 22:54:31.074112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.070789    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q" event={"ID":"c26fda6b-df94-40cd-9b24-d6aad8d90390","Type":"ContainerDied","Data":"5433c076f8accb9ae0c7b1e70e20c4668a532e7e5b6926de9790af6eb1c8d880"}
Oct 29 22:54:31.459242 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.459207    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:54:31.459519 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.459489    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:54:31.893143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.892105    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:31.919331 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.919295    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-xks-ca\") pod \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") "
Oct 29 22:54:31.919546 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.919489    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-kube-api-access-97bc7\" (UniqueName: \"kubernetes.io/projected/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-kube-api-access-97bc7\") pod \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") "
Oct 29 22:54:31.919546 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.919533    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-openshift-ca\") pod \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\" (UID: \"b603f0e6-fc00-410d-9a9b-bd186e3ff01c\") "
Oct 29 22:54:31.920237 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.920194    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-openshift-ca" pod "b603f0e6-fc00-410d-9a9b-bd186e3ff01c" (UID: "b603f0e6-fc00-410d-9a9b-bd186e3ff01c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:31.920424 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.920400    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:31.922633 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.922599    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-xks-ca" pod "b603f0e6-fc00-410d-9a9b-bd186e3ff01c" (UID: "b603f0e6-fc00-410d-9a9b-bd186e3ff01c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:31.922838 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:31.922626    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-kube-api-access-97bc7" pod "b603f0e6-fc00-410d-9a9b-bd186e3ff01c" (UID: "b603f0e6-fc00-410d-9a9b-bd186e3ff01c"). InnerVolumeSpecName "kube-api-access-97bc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:32.026562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.023902    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:32.026562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.023937    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97bc7\" (UniqueName: \"kubernetes.io/projected/b603f0e6-fc00-410d-9a9b-bd186e3ff01c-kube-api-access-97bc7\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:32.096151 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.096058    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"
Oct 29 22:54:32.096151 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.096109    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm" event={"ID":"b603f0e6-fc00-410d-9a9b-bd186e3ff01c","Type":"ContainerDied","Data":"34d107d575074dab5e5418a5aa8217995383b4b080b50288081b9924652f451f"}
Oct 29 22:54:32.096339 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.096173    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34d107d575074dab5e5418a5aa8217995383b4b080b50288081b9924652f451f"
Oct 29 22:54:32.273809 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.273786    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:32.331419 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.331382    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-openshift-ca\") pod \"c26fda6b-df94-40cd-9b24-d6aad8d90390\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") "
Oct 29 22:54:32.331597 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.331434    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-xks-ca\") pod \"c26fda6b-df94-40cd-9b24-d6aad8d90390\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") "
Oct 29 22:54:32.331597 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.331533    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26fda6b-df94-40cd-9b24-d6aad8d90390-kube-api-access-hm4xq\" (UniqueName: \"kubernetes.io/projected/c26fda6b-df94-40cd-9b24-d6aad8d90390-kube-api-access-hm4xq\") pod \"c26fda6b-df94-40cd-9b24-d6aad8d90390\" (UID: \"c26fda6b-df94-40cd-9b24-d6aad8d90390\") "
Oct 29 22:54:32.331807 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.331779    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-openshift-ca" pod "c26fda6b-df94-40cd-9b24-d6aad8d90390" (UID: "c26fda6b-df94-40cd-9b24-d6aad8d90390"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:32.332017 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.331995    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:32.334217 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.334188    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-xks-ca" pod "c26fda6b-df94-40cd-9b24-d6aad8d90390" (UID: "c26fda6b-df94-40cd-9b24-d6aad8d90390"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:32.334356 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.334283    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26fda6b-df94-40cd-9b24-d6aad8d90390-kube-api-access-hm4xq" pod "c26fda6b-df94-40cd-9b24-d6aad8d90390" (UID: "c26fda6b-df94-40cd-9b24-d6aad8d90390"). InnerVolumeSpecName "kube-api-access-hm4xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:32.433315 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.433218    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4xq\" (UniqueName: \"kubernetes.io/projected/c26fda6b-df94-40cd-9b24-d6aad8d90390-kube-api-access-hm4xq\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:32.433315 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.433246    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fda6b-df94-40cd-9b24-d6aad8d90390-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:32.456928 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.456886    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:54:32.457120 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:32.456946    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 29 22:54:33.112098 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:33.111992    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"
Oct 29 22:54:33.112098 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:33.112011    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q" event={"ID":"c26fda6b-df94-40cd-9b24-d6aad8d90390","Type":"ContainerDied","Data":"5864bdc215a5cf7e99b44d3922118946b189a7b506d31f3cdb84f615f6ba3b72"}
Oct 29 22:54:33.112671 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:33.112118    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5864bdc215a5cf7e99b44d3922118946b189a7b506d31f3cdb84f615f6ba3b72"
Oct 29 22:54:33.693541 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:33.693507    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:33.694246 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:33.694220    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:33.695137 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:33.695118    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 22:54:40.252982 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.252952    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"]
Oct 29 22:54:40.253451 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.253435    4742 cpu_manager.go:414] "containerMap: removing container" podUID="b603f0e6-fc00-410d-9a9b-bd186e3ff01c" containerName="revoke-keys"
Oct 29 22:54:40.253451 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.253451    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b603f0e6-fc00-410d-9a9b-bd186e3ff01c" containerName="revoke-keys"
Oct 29 22:54:40.253539 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.253480    4742 cpu_manager.go:414] "containerMap: removing container" podUID="c26fda6b-df94-40cd-9b24-d6aad8d90390" containerName="revoke-keys"
Oct 29 22:54:40.253539 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.253487    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26fda6b-df94-40cd-9b24-d6aad8d90390" containerName="revoke-keys"
Oct 29 22:54:40.253599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.253564    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26fda6b-df94-40cd-9b24-d6aad8d90390" containerName="revoke-keys"
Oct 29 22:54:40.253599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.253574    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="b603f0e6-fc00-410d-9a9b-bd186e3ff01c" containerName="revoke-keys"
Oct 29 22:54:40.256643 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.256628    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.275551 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.275523    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"]
Oct 29 22:54:40.284161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.284134    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:40.284336 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.284241    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:54:40.359290 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.359255    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.359465 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.359412    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.359507 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.359473    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4khm\" (UniqueName: \"kubernetes.io/projected/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-kube-api-access-m4khm\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.460169 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.460133    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.460355 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.460222    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4khm\" (UniqueName: \"kubernetes.io/projected/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-kube-api-access-m4khm\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.460355 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.460299    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.460792 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.460773    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.462680 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.462658    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.479873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.479843    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4khm\" (UniqueName: \"kubernetes.io/projected/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-kube-api-access-m4khm\") pod \"maas-api-revoke-sub-3d99cc79-sh62j\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.569058 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.568982    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:40.703737 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.703700    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"]
Oct 29 22:54:40.708510 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.708487    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.741247 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.741177    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a2e7c8-openshift-default-dockercfg-lq5n9\""
Oct 29 22:54:40.743759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.743728    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"]
Oct 29 22:54:40.763796 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.763765    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.764034 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.763919    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.764145 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.764118    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.764242 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.764161    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.764307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.764249    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.764362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.764306    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.764362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.764336    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7j6p\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-kube-api-access-x7j6p\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.764469 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.764452    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.764509 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.764483    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.865571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865533    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.865754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865577    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.865754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865610    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.865754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865725    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.865931 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865818    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.865931 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865841    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.865931 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865881    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.866108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865932    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.866108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.865957    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7j6p\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-kube-api-access-x7j6p\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.866108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.866008    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.866540 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.866513    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.866715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.866691    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.867029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.867006    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.867157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.867138    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.867418 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.867395    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"]
Oct 29 22:54:40.868365 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.868334    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.868617 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.868589    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.869795 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:54:40.869767    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7b19ca_cd3d_4ffa_8dca_ad476791b4d1.slice/crio-904bd8b40c930fe6bfb1ec74568946f886eeef4908ee58a4b4112f2eeb8c04c8 WatchSource:0}: Error finding container 904bd8b40c930fe6bfb1ec74568946f886eeef4908ee58a4b4112f2eeb8c04c8: Status 404 returned error can't find the container with id 904bd8b40c930fe6bfb1ec74568946f886eeef4908ee58a4b4112f2eeb8c04c8
Oct 29 22:54:40.890045 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.890020    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7j6p\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-kube-api-access-x7j6p\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:40.891727 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:40.891693    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:41.022209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:41.022177    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:41.154406 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:41.154308    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j" event={"ID":"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1","Type":"ContainerStarted","Data":"904bd8b40c930fe6bfb1ec74568946f886eeef4908ee58a4b4112f2eeb8c04c8"}
Oct 29 22:54:41.343920 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:41.343892    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"]
Oct 29 22:54:41.345948 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:54:41.345920    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3eee93_3339_4aab_aa07_b7afcf5a56ca.slice/crio-0d49cc8561a281a1f5ed2d2f3cfe7439b3afeeaaae70b38533ea4fbd7c773405 WatchSource:0}: Error finding container 0d49cc8561a281a1f5ed2d2f3cfe7439b3afeeaaae70b38533ea4fbd7c773405: Status 404 returned error can't find the container with id 0d49cc8561a281a1f5ed2d2f3cfe7439b3afeeaaae70b38533ea4fbd7c773405
Oct 29 22:54:41.348488 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:41.348455    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:41.348590 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:41.348525    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:41.348590 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:41.348557    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:54:42.193124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:42.193057    4742 generic.go:356] "Generic (PLEG): container finished" podID="5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1" containerID="b67b21c45457484d74b7b5e602aa482995ddd181e888a3bb0e9f1f12151c5495" exitCode=0
Oct 29 22:54:42.193513 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:42.193487    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j" event={"ID":"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1","Type":"ContainerDied","Data":"b67b21c45457484d74b7b5e602aa482995ddd181e888a3bb0e9f1f12151c5495"}
Oct 29 22:54:42.195389 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:42.195359    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" event={"ID":"8b3eee93-3339-4aab-aa07-b7afcf5a56ca","Type":"ContainerStarted","Data":"613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349"}
Oct 29 22:54:42.195489 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:42.195411    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" event={"ID":"8b3eee93-3339-4aab-aa07-b7afcf5a56ca","Type":"ContainerStarted","Data":"0d49cc8561a281a1f5ed2d2f3cfe7439b3afeeaaae70b38533ea4fbd7c773405"}
Oct 29 22:54:42.280109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:42.279374    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" podStartSLOduration=2.279355004 podStartE2EDuration="2.279355004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:42.276989693 +0000 UTC m=+739.417731278" watchObservedRunningTime="2026-10-29 22:54:42.279355004 +0000 UTC m=+739.420096591"
Oct 29 22:54:43.022472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.022439    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:43.023640 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.023616    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:54:43.023757 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.023668    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:54:43.411940 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.411916    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:43.507602 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.507569    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-openshift-ca\") pod \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") "
Oct 29 22:54:43.507800 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.507647    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-kube-api-access-m4khm\" (UniqueName: \"kubernetes.io/projected/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-kube-api-access-m4khm\") pod \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") "
Oct 29 22:54:43.507800 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.507700    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-xks-ca\") pod \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\" (UID: \"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1\") "
Oct 29 22:54:43.508029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.508005    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-openshift-ca" pod "5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1" (UID: "5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:43.508226 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.508206    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:43.509850 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.509821    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-xks-ca" pod "5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1" (UID: "5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:43.509947 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.509881    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-kube-api-access-m4khm" pod "5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1" (UID: "5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1"). InnerVolumeSpecName "kube-api-access-m4khm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:43.608129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.606756    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"]
Oct 29 22:54:43.608129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.607835    4742 cpu_manager.go:414] "containerMap: removing container" podUID="5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1" containerName="revoke-keys"
Oct 29 22:54:43.608129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.607856    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1" containerName="revoke-keys"
Oct 29 22:54:43.608129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.608011    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1" containerName="revoke-keys"
Oct 29 22:54:43.615110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.609441    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4khm\" (UniqueName: \"kubernetes.io/projected/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-kube-api-access-m4khm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:43.615539 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.615517    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:43.615666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.614402    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"]
Oct 29 22:54:43.625144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.611997    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.723119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.722696    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.723119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.722745    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbpn\" (UniqueName: \"kubernetes.io/projected/b443449d-5949-4f91-bb46-9e786c419e0b-kube-api-access-5xbpn\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.723119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.722831    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.823834 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.823797    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.824012 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.823977    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.824012 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.824006    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbpn\" (UniqueName: \"kubernetes.io/projected/b443449d-5949-4f91-bb46-9e786c419e0b-kube-api-access-5xbpn\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.824521 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.824496    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.826391 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.826369    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:43.842661 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:43.842636    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbpn\" (UniqueName: \"kubernetes.io/projected/b443449d-5949-4f91-bb46-9e786c419e0b-kube-api-access-5xbpn\") pod \"maas-api-revoke-sub-fdf3b816-2kdxx\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:44.032156 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:44.030760    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:54:44.032156 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:44.030823    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:54:44.032156 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:44.031398    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:44.293168 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:44.291656    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"
Oct 29 22:54:44.321110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:44.304167    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j" event={"ID":"5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1","Type":"ContainerDied","Data":"904bd8b40c930fe6bfb1ec74568946f886eeef4908ee58a4b4112f2eeb8c04c8"}
Oct 29 22:54:44.321110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:44.304244    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="904bd8b40c930fe6bfb1ec74568946f886eeef4908ee58a4b4112f2eeb8c04c8"
Oct 29 22:54:45.100161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.076266    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:54:45.100161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.076326    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:54:45.106508 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.106464    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"]
Oct 29 22:54:45.468274 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.468204    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:45.567122 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.550073    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx" event={"ID":"b443449d-5949-4f91-bb46-9e786c419e0b","Type":"ContainerStarted","Data":"36ca177a1042cbc2e21185796736986f6211ffb636afe08a722ff2ed13ebe5e5"}
Oct 29 22:54:45.567122 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.550208    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:45.605129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.602196    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:45.691353 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.690187    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:45.720276 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.719166    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:54:45.744692 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.743187    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:45.825110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.824183    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:45.851109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:45.850751    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:54:46.031451 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.031355    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:54:46.031719 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.031689    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:54:46.471407 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.471361    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:46.472733 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.472710    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:54:46.551687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.551638    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:46.552304 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.552278    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:54:46.735420 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.735325    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:46.736375 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.736352    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:54:46.981616 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:46.981563    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.048052 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.047965    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:54:47.107144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.095712    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:54:47.107144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.095777    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:54:47.220350 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.220264    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.373426 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.373374    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.476629 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.476586    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx" event={"ID":"b443449d-5949-4f91-bb46-9e786c419e0b","Type":"ContainerStarted","Data":"feef846888f25ec00dc6ef585b79dc9ef1b9d682ae7d8c7c3a5a2ad59d555235"}
Oct 29 22:54:47.477527 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.477492    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.492248 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.492203    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.597131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.591230    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:54:47.597131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.596385    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.618240 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.614171    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.742322 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.742210    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.788645 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.788603    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:54:47.789642 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.789606    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:47.991461 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.991421    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:54:47.991706 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:47.991679    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:48.026830 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.026735    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 29 22:54:48.027077 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.027047    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 29 22:54:48.037232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.037183    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:54:48.037487 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.037451    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:48.051306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.051267    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:48.109108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.108440    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:54:48.221275 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.221236    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:54:48.224361 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.224337    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:54:48.254268 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.254242    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:54:48.279510 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.271313    4742 generic.go:356] "Generic (PLEG): container finished" podID="b443449d-5949-4f91-bb46-9e786c419e0b" containerID="feef846888f25ec00dc6ef585b79dc9ef1b9d682ae7d8c7c3a5a2ad59d555235" exitCode=0
Oct 29 22:54:48.279510 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.272258    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx" event={"ID":"b443449d-5949-4f91-bb46-9e786c419e0b","Type":"ContainerDied","Data":"feef846888f25ec00dc6ef585b79dc9ef1b9d682ae7d8c7c3a5a2ad59d555235"}
Oct 29 22:54:48.296336 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.296263    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx" podStartSLOduration=5.296241609 podStartE2EDuration="5.296241609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:47.591185011 +0000 UTC m=+744.731926597" watchObservedRunningTime="2026-10-29 22:54:48.296241609 +0000 UTC m=+745.436983194"
Oct 29 22:54:48.303124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:48.302912    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:54:49.202384 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.202336    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:49.202868 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.202745    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:49.203828 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.203804    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 22:54:49.478572 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.478541    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:49.520391 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.520361    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-openshift-ca\") pod \"b443449d-5949-4f91-bb46-9e786c419e0b\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") "
Oct 29 22:54:49.520571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.520474    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b443449d-5949-4f91-bb46-9e786c419e0b-kube-api-access-5xbpn\" (UniqueName: \"kubernetes.io/projected/b443449d-5949-4f91-bb46-9e786c419e0b-kube-api-access-5xbpn\") pod \"b443449d-5949-4f91-bb46-9e786c419e0b\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") "
Oct 29 22:54:49.520655 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.520635    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-xks-ca\") pod \"b443449d-5949-4f91-bb46-9e786c419e0b\" (UID: \"b443449d-5949-4f91-bb46-9e786c419e0b\") "
Oct 29 22:54:49.520786 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.520758    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-openshift-ca" pod "b443449d-5949-4f91-bb46-9e786c419e0b" (UID: "b443449d-5949-4f91-bb46-9e786c419e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:49.521114 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.521075    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:49.528506 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.528476    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-xks-ca" pod "b443449d-5949-4f91-bb46-9e786c419e0b" (UID: "b443449d-5949-4f91-bb46-9e786c419e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:49.528506 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.528494    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b443449d-5949-4f91-bb46-9e786c419e0b-kube-api-access-5xbpn" pod "b443449d-5949-4f91-bb46-9e786c419e0b" (UID: "b443449d-5949-4f91-bb46-9e786c419e0b"). InnerVolumeSpecName "kube-api-access-5xbpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:49.622284 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.622255    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b443449d-5949-4f91-bb46-9e786c419e0b-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:49.622284 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:49.622280    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xbpn\" (UniqueName: \"kubernetes.io/projected/b443449d-5949-4f91-bb46-9e786c419e0b-kube-api-access-5xbpn\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:50.284253 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:50.284220    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"
Oct 29 22:54:50.284253 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:50.284251    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx" event={"ID":"b443449d-5949-4f91-bb46-9e786c419e0b","Type":"ContainerDied","Data":"36ca177a1042cbc2e21185796736986f6211ffb636afe08a722ff2ed13ebe5e5"}
Oct 29 22:54:50.284828 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:50.284294    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ca177a1042cbc2e21185796736986f6211ffb636afe08a722ff2ed13ebe5e5"
Oct 29 22:54:52.251326 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.251285    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"]
Oct 29 22:54:52.251971 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.251948    4742 cpu_manager.go:414] "containerMap: removing container" podUID="b443449d-5949-4f91-bb46-9e786c419e0b" containerName="revoke-keys"
Oct 29 22:54:52.252051 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.251972    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b443449d-5949-4f91-bb46-9e786c419e0b" containerName="revoke-keys"
Oct 29 22:54:52.252140 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.252127    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="b443449d-5949-4f91-bb46-9e786c419e0b" containerName="revoke-keys"
Oct 29 22:54:52.255631 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.255610    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.281959 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.281933    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:52.282142 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.282050    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:54:52.285031 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.285005    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"]
Oct 29 22:54:52.353980 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.353946    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.353980 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.353979    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwt8l\" (UniqueName: \"kubernetes.io/projected/62950ab1-2f2c-4966-99d5-08cc77b16da8-kube-api-access-qwt8l\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.354281 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.354256    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.455825 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.455790    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.455974 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.455829    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwt8l\" (UniqueName: \"kubernetes.io/projected/62950ab1-2f2c-4966-99d5-08cc77b16da8-kube-api-access-qwt8l\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.455974 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.455898    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.456598 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.456574    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.458228 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.458207    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.476048 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.476017    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwt8l\" (UniqueName: \"kubernetes.io/projected/62950ab1-2f2c-4966-99d5-08cc77b16da8-kube-api-access-qwt8l\") pod \"maas-api-revoke-sub-592c9d23-lqfxq\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.568052 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.567964    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:52.872936 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:52.872913    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"]
Oct 29 22:54:52.875025 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:54:52.874996    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62950ab1_2f2c_4966_99d5_08cc77b16da8.slice/crio-f8f6129474884c19e0c454cb766bccd8c4e43e66cfb40406728836529ef20ead WatchSource:0}: Error finding container f8f6129474884c19e0c454cb766bccd8c4e43e66cfb40406728836529ef20ead: Status 404 returned error can't find the container with id f8f6129474884c19e0c454cb766bccd8c4e43e66cfb40406728836529ef20ead
Oct 29 22:54:53.182677 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:54:53.182639    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:54:53.302604 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:53.302567    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq" event={"ID":"62950ab1-2f2c-4966-99d5-08cc77b16da8","Type":"ContainerStarted","Data":"1ebb403fa4e892f801e01619b34be43f604604c4f4d533df4cb0b3757e7f563e"}
Oct 29 22:54:53.302998 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:53.302652    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq" event={"ID":"62950ab1-2f2c-4966-99d5-08cc77b16da8","Type":"ContainerStarted","Data":"f8f6129474884c19e0c454cb766bccd8c4e43e66cfb40406728836529ef20ead"}
Oct 29 22:54:53.343349 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:53.343289    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq" podStartSLOduration=1.3432749529999999 podStartE2EDuration="1.343274953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:53.340289182 +0000 UTC m=+750.481030766" watchObservedRunningTime="2026-10-29 22:54:53.343274953 +0000 UTC m=+750.484016536"
Oct 29 22:54:54.309669 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:54.309635    4742 generic.go:356] "Generic (PLEG): container finished" podID="62950ab1-2f2c-4966-99d5-08cc77b16da8" containerID="1ebb403fa4e892f801e01619b34be43f604604c4f4d533df4cb0b3757e7f563e" exitCode=0
Oct 29 22:54:54.310167 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:54.309735    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq" event={"ID":"62950ab1-2f2c-4966-99d5-08cc77b16da8","Type":"ContainerDied","Data":"1ebb403fa4e892f801e01619b34be43f604604c4f4d533df4cb0b3757e7f563e"}
Oct 29 22:54:55.452756 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.452732    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:55.489262 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.489232    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-xks-ca\") pod \"62950ab1-2f2c-4966-99d5-08cc77b16da8\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") "
Oct 29 22:54:55.489396 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.489308    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62950ab1-2f2c-4966-99d5-08cc77b16da8-kube-api-access-qwt8l\" (UniqueName: \"kubernetes.io/projected/62950ab1-2f2c-4966-99d5-08cc77b16da8-kube-api-access-qwt8l\") pod \"62950ab1-2f2c-4966-99d5-08cc77b16da8\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") "
Oct 29 22:54:55.489396 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.489371    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-openshift-ca\") pod \"62950ab1-2f2c-4966-99d5-08cc77b16da8\" (UID: \"62950ab1-2f2c-4966-99d5-08cc77b16da8\") "
Oct 29 22:54:55.489809 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.489780    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-openshift-ca" pod "62950ab1-2f2c-4966-99d5-08cc77b16da8" (UID: "62950ab1-2f2c-4966-99d5-08cc77b16da8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:55.491319 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.491293    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-xks-ca" pod "62950ab1-2f2c-4966-99d5-08cc77b16da8" (UID: "62950ab1-2f2c-4966-99d5-08cc77b16da8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:55.491319 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.491313    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62950ab1-2f2c-4966-99d5-08cc77b16da8-kube-api-access-qwt8l" pod "62950ab1-2f2c-4966-99d5-08cc77b16da8" (UID: "62950ab1-2f2c-4966-99d5-08cc77b16da8"). InnerVolumeSpecName "kube-api-access-qwt8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:55.591232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.591197    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:55.591232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.591227    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwt8l\" (UniqueName: \"kubernetes.io/projected/62950ab1-2f2c-4966-99d5-08cc77b16da8-kube-api-access-qwt8l\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:55.591232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:55.591237    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62950ab1-2f2c-4966-99d5-08cc77b16da8-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:54:56.322032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:56.321994    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq" event={"ID":"62950ab1-2f2c-4966-99d5-08cc77b16da8","Type":"ContainerDied","Data":"f8f6129474884c19e0c454cb766bccd8c4e43e66cfb40406728836529ef20ead"}
Oct 29 22:54:56.322032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:56.322016    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"
Oct 29 22:54:56.322270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:54:56.322059    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f6129474884c19e0c454cb766bccd8c4e43e66cfb40406728836529ef20ead"
Oct 29 22:55:02.786255 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:02.786206    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:02.786879 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:02.786844    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-c7pw8" podUID="87c1ed81-d887-47a9-a02e-04da08ca80b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.854030 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:02.853976    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:02.854371 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:02.854323    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fbvvd" podUID="25ac9a2a-6abb-4ddb-aa72-b38f589e13b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.868240 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:02.868197    4742 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:02.868485 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:02.868455    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-sk7lx" podUID="07991432-059a-40a2-9295-8d113f364302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:05.159058 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.159023    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"]
Oct 29 22:55:05.159575 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.159558    4742 cpu_manager.go:414] "containerMap: removing container" podUID="62950ab1-2f2c-4966-99d5-08cc77b16da8" containerName="revoke-keys"
Oct 29 22:55:05.159619 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.159576    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62950ab1-2f2c-4966-99d5-08cc77b16da8" containerName="revoke-keys"
Oct 29 22:55:05.159663 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.159655    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="62950ab1-2f2c-4966-99d5-08cc77b16da8" containerName="revoke-keys"
Oct 29 22:55:05.165945 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.165925    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.184538 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.184508    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"]
Oct 29 22:55:05.190372 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.190341    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:05.190579 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.190420    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:55:05.279959 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.279924    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.280205 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.279985    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.280205 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.280173    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59wvr\" (UniqueName: \"kubernetes.io/projected/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-kube-api-access-59wvr\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.380859 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.380819    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.381036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.380908    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59wvr\" (UniqueName: \"kubernetes.io/projected/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-kube-api-access-59wvr\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.381036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.381012    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.381634 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.381611    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.383196 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.383172    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.400778 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.400748    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59wvr\" (UniqueName: \"kubernetes.io/projected/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-kube-api-access-59wvr\") pod \"maas-api-revoke-sub-c16610c5-8h9cl\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.480915 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.480835    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:05.781283 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:05.781255    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"]
Oct 29 22:55:05.783218 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:55:05.783187    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ea7194_9f8a_45e8_8660_b3cb2ebea00c.slice/crio-c133fdaf262964340d99b608d1afc11d6342d10b58e1d659193305a3b7d68615 WatchSource:0}: Error finding container c133fdaf262964340d99b608d1afc11d6342d10b58e1d659193305a3b7d68615: Status 404 returned error can't find the container with id c133fdaf262964340d99b608d1afc11d6342d10b58e1d659193305a3b7d68615
Oct 29 22:55:06.430670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:06.430631    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl" event={"ID":"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c","Type":"ContainerStarted","Data":"e177045e26e70030e345e97dd884c8fb320646e2cb85675321c3886c3ada3611"}
Oct 29 22:55:06.431151 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:06.430688    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl" event={"ID":"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c","Type":"ContainerStarted","Data":"c133fdaf262964340d99b608d1afc11d6342d10b58e1d659193305a3b7d68615"}
Oct 29 22:55:06.471335 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:06.471274    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl" podStartSLOduration=1.4712558869999999 podStartE2EDuration="1.471255887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:06.468577666 +0000 UTC m=+763.609319251" watchObservedRunningTime="2026-10-29 22:55:06.471255887 +0000 UTC m=+763.611997528"
Oct 29 22:55:07.437951 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:07.437908    4742 generic.go:356] "Generic (PLEG): container finished" podID="a5ea7194-9f8a-45e8-8660-b3cb2ebea00c" containerID="e177045e26e70030e345e97dd884c8fb320646e2cb85675321c3886c3ada3611" exitCode=0
Oct 29 22:55:07.438423 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:07.437965    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl" event={"ID":"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c","Type":"ContainerDied","Data":"e177045e26e70030e345e97dd884c8fb320646e2cb85675321c3886c3ada3611"}
Oct 29 22:55:08.613240 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.613210    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:08.714172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.714042    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-openshift-ca\") pod \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") "
Oct 29 22:55:08.714172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.714103    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-kube-api-access-59wvr\" (UniqueName: \"kubernetes.io/projected/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-kube-api-access-59wvr\") pod \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") "
Oct 29 22:55:08.714410 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.714198    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-xks-ca\") pod \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\" (UID: \"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c\") "
Oct 29 22:55:08.714519 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.714494    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-openshift-ca" pod "a5ea7194-9f8a-45e8-8660-b3cb2ebea00c" (UID: "a5ea7194-9f8a-45e8-8660-b3cb2ebea00c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:08.716242 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.716212    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-xks-ca" pod "a5ea7194-9f8a-45e8-8660-b3cb2ebea00c" (UID: "a5ea7194-9f8a-45e8-8660-b3cb2ebea00c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:08.716350 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.716293    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-kube-api-access-59wvr" pod "a5ea7194-9f8a-45e8-8660-b3cb2ebea00c" (UID: "a5ea7194-9f8a-45e8-8660-b3cb2ebea00c"). InnerVolumeSpecName "kube-api-access-59wvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:08.814952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.814916    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:55:08.814952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.814949    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:55:08.814952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:08.814960    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59wvr\" (UniqueName: \"kubernetes.io/projected/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c-kube-api-access-59wvr\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:55:09.451832 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:09.451804    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"
Oct 29 22:55:09.452007 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:09.451832    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl" event={"ID":"a5ea7194-9f8a-45e8-8660-b3cb2ebea00c","Type":"ContainerDied","Data":"c133fdaf262964340d99b608d1afc11d6342d10b58e1d659193305a3b7d68615"}
Oct 29 22:55:09.452007 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:09.451881    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c133fdaf262964340d99b608d1afc11d6342d10b58e1d659193305a3b7d68615"
Oct 29 22:55:10.724378 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.724347    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"]
Oct 29 22:55:10.724898 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.724868    4742 cpu_manager.go:414] "containerMap: removing container" podUID="a5ea7194-9f8a-45e8-8660-b3cb2ebea00c" containerName="revoke-keys"
Oct 29 22:55:10.724898 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.724880    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ea7194-9f8a-45e8-8660-b3cb2ebea00c" containerName="revoke-keys"
Oct 29 22:55:10.725034 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.724965    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ea7194-9f8a-45e8-8660-b3cb2ebea00c" containerName="revoke-keys"
Oct 29 22:55:10.728108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.728072    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.731206 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.731175    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-dshm\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.731340 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.731241    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5tt\" (UniqueName: \"kubernetes.io/projected/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-kube-api-access-hn5tt\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.731340 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.731290    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-model-cache\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.731456 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.731382    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-home\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.731517 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.731457    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tmp-dir\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.731574 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.731547    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tls-certs\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.747641 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.747609    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"]
Oct 29 22:55:10.756032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.755994    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"distinct-w6-81f92d-kserve-self-signed-certs\""
Oct 29 22:55:10.832164 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832132    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5tt\" (UniqueName: \"kubernetes.io/projected/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-kube-api-access-hn5tt\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.832380 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832175    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-model-cache\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.832380 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832207    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-home\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.832380 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832265    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tmp-dir\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.832380 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832298    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tls-certs\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.832647 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832452    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-dshm\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.832647 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832616    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-model-cache\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.832720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832657    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-home\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.832756 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.832716    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tmp-dir\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.835057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.835028    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-dshm\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.835188 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.835097    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tls-certs\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:10.853635 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:10.853605    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5tt\" (UniqueName: \"kubernetes.io/projected/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-kube-api-access-hn5tt\") pod \"distinct-w6-81f92d-kserve-7ccb58489f-lzgng\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:11.041450 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.041360    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:11.111890 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.111860    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"]
Oct 29 22:55:11.115697 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.115673    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.134710 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.134673    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.134886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.134752    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.134886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.134795    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.134886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.134851    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.134886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.134884    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.135157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.134911    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.135157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.134969    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.135157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.135112    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcca471e-1edf-4fbe-9802-7d83169bf764-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.135157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.135144    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6chzg\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-kube-api-access-6chzg\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.142230 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.141999    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"]
Oct 29 22:55:11.235849 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.235811    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chzg\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-kube-api-access-6chzg\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236025 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.235860    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236025 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.235882    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236025 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236002    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236175 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236054    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236175 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236139    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236283 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236184    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236283 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236255    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236381 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236325    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236444 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236330    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236444 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236417    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236494    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcca471e-1edf-4fbe-9802-7d83169bf764-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.236571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.236500    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.237073 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.237051    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcca471e-1edf-4fbe-9802-7d83169bf764-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.238467 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.238447    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.238670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.238653    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.255418 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.255389    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chzg\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-kube-api-access-6chzg\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.255563 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.255416    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.340545 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.340512    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"]
Oct 29 22:55:11.341938 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:55:11.341913    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02aea966_e9cc_4ee1_93b8_5f6c173b4d91.slice/crio-08e0086ad617b072125f966456c506a8895b52bc53dbe10952811cd613f7f636 WatchSource:0}: Error finding container 08e0086ad617b072125f966456c506a8895b52bc53dbe10952811cd613f7f636: Status 404 returned error can't find the container with id 08e0086ad617b072125f966456c506a8895b52bc53dbe10952811cd613f7f636
Oct 29 22:55:11.432576 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.432355    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:11.466172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.466135    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" event={"ID":"02aea966-e9cc-4ee1-93b8-5f6c173b4d91","Type":"ContainerStarted","Data":"12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686"}
Oct 29 22:55:11.466357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.466186    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" event={"ID":"02aea966-e9cc-4ee1-93b8-5f6c173b4d91","Type":"ContainerStarted","Data":"08e0086ad617b072125f966456c506a8895b52bc53dbe10952811cd613f7f636"}
Oct 29 22:55:11.513781 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.513721    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podStartSLOduration=1.513705409 podStartE2EDuration="1.513705409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:11.512298306 +0000 UTC m=+768.653039892" watchObservedRunningTime="2026-10-29 22:55:11.513705409 +0000 UTC m=+768.654446993"
Oct 29 22:55:11.740246 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.740221    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"]
Oct 29 22:55:11.740635 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:55:11.740543    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcca471e_1edf_4fbe_9802_7d83169bf764.slice/crio-4593ceeeb8bca889dc39aa1662cd9f09bee9825d629c5314f10b95908c149d08 WatchSource:0}: Error finding container 4593ceeeb8bca889dc39aa1662cd9f09bee9825d629c5314f10b95908c149d08: Status 404 returned error can't find the container with id 4593ceeeb8bca889dc39aa1662cd9f09bee9825d629c5314f10b95908c149d08
Oct 29 22:55:11.743041 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.743009    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:11.743140 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.743080    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:11.743191 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:11.743162    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:12.041812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:12.041708    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:12.485544 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:12.485490    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" event={"ID":"dcca471e-1edf-4fbe-9802-7d83169bf764","Type":"ContainerStarted","Data":"199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116"}
Oct 29 22:55:12.485746 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:12.485588    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" event={"ID":"dcca471e-1edf-4fbe-9802-7d83169bf764","Type":"ContainerStarted","Data":"4593ceeeb8bca889dc39aa1662cd9f09bee9825d629c5314f10b95908c149d08"}
Oct 29 22:55:12.527048 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:12.526994    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" podStartSLOduration=1.526978728 podStartE2EDuration="1.526978728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:12.526244028 +0000 UTC m=+769.666985614" watchObservedRunningTime="2026-10-29 22:55:12.526978728 +0000 UTC m=+769.667720313"
Oct 29 22:55:13.432997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:13.432963    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:13.434588 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:13.434556    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:13.434720 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:13.434608    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 22:55:14.433398 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:14.433360    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:14.433813 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:14.433421    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 22:55:15.433514 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:15.433478    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:15.433938 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:15.433548    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 22:55:16.439112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:16.437317    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:16.439112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:16.437391    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 22:55:17.433417 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:17.433322    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:17.433417 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:17.433396    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 22:55:18.433729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.433690    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:18.434255 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.433758    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 22:55:18.657343 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.657301    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"]
Oct 29 22:55:18.662957 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.662928    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.681327 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.681294    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"]
Oct 29 22:55:18.691180 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.691102    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:55:18.691478 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.691457    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:18.782516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.782473    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt7vw\" (UniqueName: \"kubernetes.io/projected/077d0086-224d-4657-994b-5cbbafe2407c-kube-api-access-lt7vw\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.783026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.782940    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.783212 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.783098    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.883834 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.883786    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.884047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.883864    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.884047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.883923    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt7vw\" (UniqueName: \"kubernetes.io/projected/077d0086-224d-4657-994b-5cbbafe2407c-kube-api-access-lt7vw\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.884752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.884725    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.886267 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.886244    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.906058 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.906018    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt7vw\" (UniqueName: \"kubernetes.io/projected/077d0086-224d-4657-994b-5cbbafe2407c-kube-api-access-lt7vw\") pod \"maas-api-revoke-sub-fe1291a8-jd8qs\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:18.985124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:18.984998    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:19.294327 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:19.292772    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"]
Oct 29 22:55:19.297770 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:55:19.297247    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077d0086_224d_4657_994b_5cbbafe2407c.slice/crio-c86b3ce0b15e19ae94378a8993f0183571da0f045c266147d897d39b9ce7ab7b WatchSource:0}: Error finding container c86b3ce0b15e19ae94378a8993f0183571da0f045c266147d897d39b9ce7ab7b: Status 404 returned error can't find the container with id c86b3ce0b15e19ae94378a8993f0183571da0f045c266147d897d39b9ce7ab7b
Oct 29 22:55:19.437746 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:19.437717    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:19.438179 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:19.438012    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:19.438679 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:19.438659    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:55:19.547147 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:19.547031    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs" event={"ID":"077d0086-224d-4657-994b-5cbbafe2407c","Type":"ContainerStarted","Data":"c86b3ce0b15e19ae94378a8993f0183571da0f045c266147d897d39b9ce7ab7b"}
Oct 29 22:55:20.554581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:20.554350    4742 generic.go:356] "Generic (PLEG): container finished" podID="077d0086-224d-4657-994b-5cbbafe2407c" containerID="ad263e801eb5771d198a765dcc126724099b33474d65ba78017d0b30f4fc38a6" exitCode=0
Oct 29 22:55:20.554581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:20.554428    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs" event={"ID":"077d0086-224d-4657-994b-5cbbafe2407c","Type":"ContainerDied","Data":"ad263e801eb5771d198a765dcc126724099b33474d65ba78017d0b30f4fc38a6"}
Oct 29 22:55:21.042464 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.042430    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:21.055739 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.055709    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:21.572346 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.572316    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:55:21.729272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.729244    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:21.820024 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.819939    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077d0086-224d-4657-994b-5cbbafe2407c-kube-api-access-lt7vw\" (UniqueName: \"kubernetes.io/projected/077d0086-224d-4657-994b-5cbbafe2407c-kube-api-access-lt7vw\") pod \"077d0086-224d-4657-994b-5cbbafe2407c\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") "
Oct 29 22:55:21.820233 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.820051    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-xks-ca\") pod \"077d0086-224d-4657-994b-5cbbafe2407c\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") "
Oct 29 22:55:21.820233 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.820115    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-openshift-ca\") pod \"077d0086-224d-4657-994b-5cbbafe2407c\" (UID: \"077d0086-224d-4657-994b-5cbbafe2407c\") "
Oct 29 22:55:21.820498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.820476    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-openshift-ca" pod "077d0086-224d-4657-994b-5cbbafe2407c" (UID: "077d0086-224d-4657-994b-5cbbafe2407c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:21.822096 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.822062    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077d0086-224d-4657-994b-5cbbafe2407c-kube-api-access-lt7vw" pod "077d0086-224d-4657-994b-5cbbafe2407c" (UID: "077d0086-224d-4657-994b-5cbbafe2407c"). InnerVolumeSpecName "kube-api-access-lt7vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:21.822180 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.822101    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-xks-ca" pod "077d0086-224d-4657-994b-5cbbafe2407c" (UID: "077d0086-224d-4657-994b-5cbbafe2407c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:21.921173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.921136    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:55:21.921173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.921166    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt7vw\" (UniqueName: \"kubernetes.io/projected/077d0086-224d-4657-994b-5cbbafe2407c-kube-api-access-lt7vw\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:55:21.921173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:21.921176    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077d0086-224d-4657-994b-5cbbafe2407c-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:55:22.570617 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:22.570564    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs" event={"ID":"077d0086-224d-4657-994b-5cbbafe2407c","Type":"ContainerDied","Data":"c86b3ce0b15e19ae94378a8993f0183571da0f045c266147d897d39b9ce7ab7b"}
Oct 29 22:55:22.570800 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:22.570657    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"
Oct 29 22:55:22.570800 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:22.570672    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86b3ce0b15e19ae94378a8993f0183571da0f045c266147d897d39b9ce7ab7b"
Oct 29 22:55:23.123038 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:55:23.123001    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:55:40.168871 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.168835    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"]
Oct 29 22:55:40.169370 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.169317    4742 cpu_manager.go:414] "containerMap: removing container" podUID="077d0086-224d-4657-994b-5cbbafe2407c" containerName="revoke-keys"
Oct 29 22:55:40.169370 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.169328    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077d0086-224d-4657-994b-5cbbafe2407c" containerName="revoke-keys"
Oct 29 22:55:40.169446 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.169437    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="077d0086-224d-4657-994b-5cbbafe2407c" containerName="revoke-keys"
Oct 29 22:55:40.172577 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.172561    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.203318 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.203277    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"]
Oct 29 22:55:40.269023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.268984    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.269241 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.269048    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.269241 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.269178    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkkq\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-kube-api-access-5lkkq\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.269333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.269262    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.269333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.269291    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.269425 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.269361    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.269479 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.269438    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.269550 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.269536    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.269658 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.269635    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.370889 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.370845    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.370905    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.370929    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkkq\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-kube-api-access-5lkkq\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.370974    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.370998    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371024    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371054    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371169    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371226    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371381    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371623 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371512    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371681 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371668    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371728    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.371802 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.371756    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.373387 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.373364    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.373484 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.373475    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.396229 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.396202    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.396375 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.396346    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkkq\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-kube-api-access-5lkkq\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.486431 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.486343    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:40.804717 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.804688    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"]
Oct 29 22:55:40.806257 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:55:40.806225    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f9d83d_a5a8_4739_9769_bdc9c275ab7a.slice/crio-ecd14dabf7e940733cc0cd0a5111690ff9d53b7172c51a727ac8eff0407c8a6b WatchSource:0}: Error finding container ecd14dabf7e940733cc0cd0a5111690ff9d53b7172c51a727ac8eff0407c8a6b: Status 404 returned error can't find the container with id ecd14dabf7e940733cc0cd0a5111690ff9d53b7172c51a727ac8eff0407c8a6b
Oct 29 22:55:40.808783 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.808748    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:40.808881 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.808827    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:40.808881 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:40.808864    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:41.056752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.056673    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"]
Oct 29 22:55:41.060950 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.060926    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.087587 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.087554    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"]
Oct 29 22:55:41.180054 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180013    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.180571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180170    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.180571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180232    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.180571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180265    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.180571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180350    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.180571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180390    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9bc7\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-kube-api-access-z9bc7\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.180571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180467    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.180571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180534    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.180961 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.180580    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281450 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281410    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281631 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281464    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281631 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281491    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9bc7\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-kube-api-access-z9bc7\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281631 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281549    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281631 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281613    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281827 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281663    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281827 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281708    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281827 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281781    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.281974 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.281833    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.282029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.282009    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.282113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.282041    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.282256 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.282229    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.282377 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.282271    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.282538 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.282520    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.284031 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.284000    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.284242 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.284220    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.303072 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.303028    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.303787 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.303757    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9bc7\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-kube-api-access-z9bc7\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.377955 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.377870    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:41.742549 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.742462    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" event={"ID":"08f9d83d-a5a8-4739-9769-bdc9c275ab7a","Type":"ContainerStarted","Data":"290b21423a9e6505f52b16acb1e0bc1c49cdae1d84a720a6ae78956bb96ff1a4"}
Oct 29 22:55:41.742549 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.742514    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" event={"ID":"08f9d83d-a5a8-4739-9769-bdc9c275ab7a","Type":"ContainerStarted","Data":"ecd14dabf7e940733cc0cd0a5111690ff9d53b7172c51a727ac8eff0407c8a6b"}
Oct 29 22:55:41.754040 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.754012    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"]
Oct 29 22:55:41.754167 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:55:41.754097    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d97a9b0_9aed_43ea_90ed_d280767bc9cc.slice/crio-605dca7779426edd682d65f21471ed07ff254d41a529f3e29980e69a77718f86 WatchSource:0}: Error finding container 605dca7779426edd682d65f21471ed07ff254d41a529f3e29980e69a77718f86: Status 404 returned error can't find the container with id 605dca7779426edd682d65f21471ed07ff254d41a529f3e29980e69a77718f86
Oct 29 22:55:41.756653 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.756619    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:41.756754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.756687    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:41.756754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.756716    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:41.792526 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:41.792465    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podStartSLOduration=1.7924486 podStartE2EDuration="1.7924486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:41.790248821 +0000 UTC m=+798.930990407" watchObservedRunningTime="2026-10-29 22:55:41.7924486 +0000 UTC m=+798.933190187"
Oct 29 22:55:42.494108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:42.493943    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:42.494618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:42.494150    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:55:42.494618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:42.494202    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:55:42.749792 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:42.749705    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" event={"ID":"3d97a9b0-9aed-43ea-90ed-d280767bc9cc","Type":"ContainerStarted","Data":"794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330"}
Oct 29 22:55:42.749792 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:42.749770    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" event={"ID":"3d97a9b0-9aed-43ea-90ed-d280767bc9cc","Type":"ContainerStarted","Data":"605dca7779426edd682d65f21471ed07ff254d41a529f3e29980e69a77718f86"}
Oct 29 22:55:42.815698 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:42.815632    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" podStartSLOduration=1.815610527 podStartE2EDuration="1.815610527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:42.813195132 +0000 UTC m=+799.953936720" watchObservedRunningTime="2026-10-29 22:55:42.815610527 +0000 UTC m=+799.956352113"
Oct 29 22:55:43.381108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:43.380917    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:43.381330 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:43.381131    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:55:43.381330 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:43.381193    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 29 22:55:43.492110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:43.489538    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:55:43.492110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:43.489612    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:55:44.384111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:44.381680    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:55:44.384111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:44.381775    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 29 22:55:44.489109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:44.488564    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:55:44.489109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:44.488629    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:55:45.409416 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:45.408878    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:55:45.409416 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:45.408938    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 29 22:55:45.507258 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:45.507215    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:55:45.507438 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:45.507298    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:55:46.383111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:46.382263    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:55:46.383111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:46.382338    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 29 22:55:46.489109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:46.488503    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:55:46.489109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:46.488578    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:55:47.380135 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:47.379540    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:55:47.380135 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:47.379614    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 29 22:55:47.488646 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:47.488608    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 29 22:55:47.488940 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:47.488911    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 29 22:55:48.378762 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:48.378727    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 29 22:55:48.379173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:48.378794    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 29 22:55:48.491650 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:48.491622    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:48.492161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:48.492138    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:48.493025 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:48.493003    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 22:55:49.383519 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:49.383483    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:49.383966 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:49.383930    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:49.384695 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:49.384676    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 22:55:52.814718 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.814686    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"]
Oct 29 22:55:52.819610 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.819588    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:52.850020 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.849983    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"distinct-w3-07eef2-kserve-self-signed-certs\""
Oct 29 22:55:52.853439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.853409    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"]
Oct 29 22:55:52.986248 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.986214    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-model-cache\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:52.986248 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.986247    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tmp-dir\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:52.986509 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.986467    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tls-certs\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:52.986563 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.986523    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-home\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:52.986601 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.986588    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-dshm\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:52.986640 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:52.986622    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhzg\" (UniqueName: \"kubernetes.io/projected/1bf42312-d61e-4b09-8a1b-37acd752ce5c-kube-api-access-rhhzg\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.087841 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.087808    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tls-certs\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.087984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.087855    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-home\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.088043 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.088022    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-dshm\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.088139 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.088120    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhzg\" (UniqueName: \"kubernetes.io/projected/1bf42312-d61e-4b09-8a1b-37acd752ce5c-kube-api-access-rhhzg\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.088232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.088216    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-home\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.088286 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.088268    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-model-cache\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.088335 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.088291    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tmp-dir\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.088568 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.088543    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-model-cache\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.088677 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.088640    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tmp-dir\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.090481 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.090456    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tls-certs\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.090570 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.090461    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-dshm\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.107604 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.107565    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhzg\" (UniqueName: \"kubernetes.io/projected/1bf42312-d61e-4b09-8a1b-37acd752ce5c-kube-api-access-rhhzg\") pod \"distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.129499 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:55:53.129434    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:55:53.131784 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.131759    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:53.428406 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.428361    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"]
Oct 29 22:55:53.428618 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:55:53.428596    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bf42312_d61e_4b09_8a1b_37acd752ce5c.slice/crio-480645aeacbea5987a9e13d6456e9f326431be39b88d75f26cdd02861478cf66 WatchSource:0}: Error finding container 480645aeacbea5987a9e13d6456e9f326431be39b88d75f26cdd02861478cf66: Status 404 returned error can't find the container with id 480645aeacbea5987a9e13d6456e9f326431be39b88d75f26cdd02861478cf66
Oct 29 22:55:53.871032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.870995    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" event={"ID":"1bf42312-d61e-4b09-8a1b-37acd752ce5c","Type":"ContainerStarted","Data":"24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed"}
Oct 29 22:55:53.871032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.871043    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" event={"ID":"1bf42312-d61e-4b09-8a1b-37acd752ce5c","Type":"ContainerStarted","Data":"480645aeacbea5987a9e13d6456e9f326431be39b88d75f26cdd02861478cf66"}
Oct 29 22:55:53.910777 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:53.910717    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podStartSLOduration=1.9107036069999999 podStartE2EDuration="1.910703607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:53.909371176 +0000 UTC m=+811.050112760" watchObservedRunningTime="2026-10-29 22:55:53.910703607 +0000 UTC m=+811.051445191"
Oct 29 22:55:54.132637 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:54.132559    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:55:55.977266 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:55.977230    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"]
Oct 29 22:55:55.981471 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:55.981448    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.035557 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.035526    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"]
Oct 29 22:55:56.128974 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.128923    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"]
Oct 29 22:55:56.131784 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.131744    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.131944 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.131800    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.131944 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.131840    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2mh4\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-kube-api-access-r2mh4\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.132115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.132049    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.132192 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.132171    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.132311 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.132292    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.132403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.132385    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9b6036-f938-43b2-8519-184c45185696-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.132551 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.132522    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.132948 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.132922    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9b6036-f938-43b2-8519-184c45185696-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.133941 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.133921    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.181310 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.181275    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"]
Oct 29 22:55:56.234038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.233935    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56eace7-1ab2-4646-9298-9f652a46bfff-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.234038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.234014    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.234306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.234061    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.234306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.234125    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.234306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.234166    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2mh4\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-kube-api-access-r2mh4\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.234670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.234639    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.234864 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.234844    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.235012 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.234994    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.235178 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235154    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.235178 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.234891    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.235322 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235159    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.235583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235555    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.235670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235600    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rld\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-kube-api-access-h4rld\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.235670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235639    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.235796 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235689    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.235796 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235769    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9b6036-f938-43b2-8519-184c45185696-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.235911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235795    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.235911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235824    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.235911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235861    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.235911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235891    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9b6036-f938-43b2-8519-184c45185696-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.235911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.235899    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.236670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.236646    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.236810 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.236667    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9b6036-f938-43b2-8519-184c45185696-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.239460 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.239432    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9b6036-f938-43b2-8519-184c45185696-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.241398 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.241375    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.263697 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.263520    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2mh4\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-kube-api-access-r2mh4\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.275370 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.275311    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.304792 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.304532    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337180    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337238    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337279    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56eace7-1ab2-4646-9298-9f652a46bfff-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337332    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337424    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337458    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337510    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337546    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rld\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-kube-api-access-h4rld\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.337581    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.338050    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.338658    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.338967    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.339037    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.340110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.339622    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56eace7-1ab2-4646-9298-9f652a46bfff-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.342235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.342210    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.342778 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.342748    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.361319 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.361280    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.361909 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.361878    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rld\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-kube-api-access-h4rld\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.454370 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.454321    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:56.739013 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.738972    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"]
Oct 29 22:55:56.748783 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.748742    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:56.748905 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.748844    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:56.748961 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.748928    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:56.945859 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.945536    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" event={"ID":"ef9b6036-f938-43b2-8519-184c45185696","Type":"ContainerStarted","Data":"2dc3b77b5d27d5092cbd5a46c14f4fe6bed59b844862088c2708ca29be94867c"}
Oct 29 22:55:56.946001 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.945918    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"]
Oct 29 22:55:56.951978 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.951950    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:56.952050 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.952014    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:56.952050 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:56.952042    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:55:57.953419 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:57.953384    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" event={"ID":"ef9b6036-f938-43b2-8519-184c45185696","Type":"ContainerStarted","Data":"5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70"}
Oct 29 22:55:57.955156 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:57.955133    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" event={"ID":"c56eace7-1ab2-4646-9298-9f652a46bfff","Type":"ContainerStarted","Data":"41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f"}
Oct 29 22:55:57.955274 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:57.955177    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" event={"ID":"c56eace7-1ab2-4646-9298-9f652a46bfff","Type":"ContainerStarted","Data":"bacb188cd80b48463dbc9074d183b3bd3197af2209b549a8069e24e61403ad3f"}
Oct 29 22:55:58.013271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:58.013213    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podStartSLOduration=3.013194648 podStartE2EDuration="3.013194648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:58.011215228 +0000 UTC m=+815.151956811" watchObservedRunningTime="2026-10-29 22:55:58.013194648 +0000 UTC m=+815.153936234"
Oct 29 22:55:58.054138 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:58.054053    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" podStartSLOduration=2.054038507 podStartE2EDuration="2.054038507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:58.053069646 +0000 UTC m=+815.193811230" watchObservedRunningTime="2026-10-29 22:55:58.054038507 +0000 UTC m=+815.194780091"
Oct 29 22:55:58.308167 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:58.307629    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:55:58.311026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:58.310578    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:55:58.311026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:58.310629    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:55:58.455695 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:58.455661    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:55:58.456900 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:58.456872    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:55:58.457039 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:58.456920    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:55:59.307516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:59.307462    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:55:59.308400 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:59.308067    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:55:59.455033 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:59.454997    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:55:59.455232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:55:59.455050    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:56:00.306115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:00.305296    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:00.306115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:00.305358    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:56:00.467110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:00.466478    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:00.467110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:00.466539    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:56:01.316110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:01.309576    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:01.316110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:01.309658    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:56:01.455272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:01.455234    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:01.455426 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:01.455293    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:56:02.306685 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:02.306636    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:02.307311 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:02.307258    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:56:02.455649 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:02.455606    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:02.455844 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:02.455691    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:56:03.132162 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:03.132118    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:56:03.145569 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:03.145540    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:56:03.305109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:03.304969    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:03.305109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:03.305031    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:56:03.455240 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:03.455159    4742 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:03.455674 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:03.455651    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 29 22:56:04.054680 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:04.054649    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:56:04.305174 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:04.305035    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:04.305174 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:04.305134    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 29 22:56:04.529592 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:04.529553    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:56:04.530895 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:04.530866    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:56:04.532714 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:04.532695    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 22:56:05.310776 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:05.310746    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:56:05.311685 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:05.311202    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:56:05.312255 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:05.312236    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 22:56:12.774921 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.774878    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"]
Oct 29 22:56:12.779153 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.779129    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.808062 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.808028    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ffba57\"/\"distinct-2-w0-ffba57-kserve-self-signed-certs\""
Oct 29 22:56:12.811728 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.811700    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"]
Oct 29 22:56:12.840730 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.840655    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-home\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.840927 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.840759    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-dshm\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.840927 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.840783    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tmp-dir\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.840927 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.840874    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pmf\" (UniqueName: \"kubernetes.io/projected/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-kube-api-access-c2pmf\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.840927 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.840914    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tls-certs\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.841161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.841121    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-model-cache\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.942556 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.942521    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-home\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.942556 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.942559    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-dshm\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.942812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.942576    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tmp-dir\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.942812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.942611    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pmf\" (UniqueName: \"kubernetes.io/projected/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-kube-api-access-c2pmf\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.942812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.942734    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tls-certs\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.942994 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.942904    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-home\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.942994 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.942926    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-model-cache\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.942994 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.942972    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tmp-dir\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.943377 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.943355    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-model-cache\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.944783 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.944762    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-dshm\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.945256 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.945235    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tls-certs\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:12.962828 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:12.962794    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pmf\" (UniqueName: \"kubernetes.io/projected/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-kube-api-access-c2pmf\") pod \"distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") " pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:13.095623 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:13.095536    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:13.400553 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:13.400525    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"]
Oct 29 22:56:13.403013 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:56:13.402983    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5b6fd0a_d942_4ed0_9f95_19dc80ff7b02.slice/crio-29ffcca9eff26db3e8808bd8f0b1d68b2eb40ff6a3274e9bceca8810dcedbec3 WatchSource:0}: Error finding container 29ffcca9eff26db3e8808bd8f0b1d68b2eb40ff6a3274e9bceca8810dcedbec3: Status 404 returned error can't find the container with id 29ffcca9eff26db3e8808bd8f0b1d68b2eb40ff6a3274e9bceca8810dcedbec3
Oct 29 22:56:14.109391 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:14.109352    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" event={"ID":"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02","Type":"ContainerStarted","Data":"cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39"}
Oct 29 22:56:14.109798 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:14.109408    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" event={"ID":"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02","Type":"ContainerStarted","Data":"29ffcca9eff26db3e8808bd8f0b1d68b2eb40ff6a3274e9bceca8810dcedbec3"}
Oct 29 22:56:14.150320 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:14.150268    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podStartSLOduration=2.150253039 podStartE2EDuration="2.150253039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:14.14810563 +0000 UTC m=+831.288847211" watchObservedRunningTime="2026-10-29 22:56:14.150253039 +0000 UTC m=+831.290994661"
Oct 29 22:56:15.095920 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:15.095887    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:18.059241 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:18.059185    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:18.306238 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:18.306192    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:56:18.516125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:18.515536    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:23.095840 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:23.095803    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:23.112131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:23.112061    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:23.123696 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:56:23.123658    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:56:23.234274 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:23.234248    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:56:26.424955 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.424912    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"]
Oct 29 22:56:26.428952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.428931    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.459872 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.459841    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"]
Oct 29 22:56:26.555739 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.555704    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.555902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.555758    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef3ee10-d844-498e-be25-9f151d022caa-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.555902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.555837    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgd5d\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-kube-api-access-wgd5d\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.555902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.555888    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef3ee10-d844-498e-be25-9f151d022caa-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.556009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.555925    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.556046 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.556027    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.556146 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.556117    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.556199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.556165    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.556279 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.556261    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.657825 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.657788    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658019 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.657850    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef3ee10-d844-498e-be25-9f151d022caa-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658019 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.657897    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgd5d\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-kube-api-access-wgd5d\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658019 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.657967    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef3ee10-d844-498e-be25-9f151d022caa-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658214 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658014    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658214 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658073    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658214 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658161    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658214 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658203    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658413 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658291    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658590 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658561    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658778 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658594    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658778 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658726    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658778 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658761    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.658956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.658833    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef3ee10-d844-498e-be25-9f151d022caa-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.660422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.660399    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.660535 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.660452    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef3ee10-d844-498e-be25-9f151d022caa-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.679344 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.679280    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.679465 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.679445    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgd5d\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-kube-api-access-wgd5d\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:26.742631 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:26.742601    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:27.059004 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:27.058980    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"]
Oct 29 22:56:27.060897 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:56:27.060869    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef3ee10_d844_498e_be25_9f151d022caa.slice/crio-799aa28db88d35531419ce9527f2a3833792a045d4602bfac9e1b97e45902656 WatchSource:0}: Error finding container 799aa28db88d35531419ce9527f2a3833792a045d4602bfac9e1b97e45902656: Status 404 returned error can't find the container with id 799aa28db88d35531419ce9527f2a3833792a045d4602bfac9e1b97e45902656
Oct 29 22:56:27.063507 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:27.063476    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:56:27.063571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:27.063543    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:56:27.063608 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:27.063572    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:56:27.252402 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:27.252361    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" event={"ID":"bef3ee10-d844-498e-be25-9f151d022caa","Type":"ContainerStarted","Data":"87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe"}
Oct 29 22:56:27.252576 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:27.252431    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" event={"ID":"bef3ee10-d844-498e-be25-9f151d022caa","Type":"ContainerStarted","Data":"799aa28db88d35531419ce9527f2a3833792a045d4602bfac9e1b97e45902656"}
Oct 29 22:56:27.293887 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:27.293838    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" podStartSLOduration=1.2938222320000001 podStartE2EDuration="1.293822232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:27.292257137 +0000 UTC m=+844.432998723" watchObservedRunningTime="2026-10-29 22:56:27.293822232 +0000 UTC m=+844.434563821"
Oct 29 22:56:27.743770 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:27.743727    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:28.396345 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.396198    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"]
Oct 29 22:56:28.456116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.417354    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.456116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.440843    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"]
Oct 29 22:56:28.504209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.503545    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb392-8e28-4cce-984c-1f06108f927a-tls-certs\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.504209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.503667    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-model-cache\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.504209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.503730    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-tmp-dir\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.504209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.503881    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/dddcb392-8e28-4cce-984c-1f06108f927a-kube-api-access-gb722\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.504209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.503969    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-home\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.504209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.503997    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-dshm\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.511220 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.511139    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"distinct-2-w3-07eef2-kserve-self-signed-certs\""
Oct 29 22:56:28.617215 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.617163    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-model-cache\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.627425 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.627398    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-tmp-dir\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.627785 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.627764    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/dddcb392-8e28-4cce-984c-1f06108f927a-kube-api-access-gb722\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.627986 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.627972    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-home\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.628097 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.628068    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-dshm\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.628250 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.628236    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb392-8e28-4cce-984c-1f06108f927a-tls-certs\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.630056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.627299    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-model-cache\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.630056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.629563    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-tmp-dir\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.630056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.629785    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-home\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.641863 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.641834    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb392-8e28-4cce-984c-1f06108f927a-tls-certs\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.666723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.648357    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-dshm\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.720993 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.720914    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/dddcb392-8e28-4cce-984c-1f06108f927a-kube-api-access-gb722\") pod \"distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") " pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:28.749943 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.749554    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:56:28.749943 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.749626    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:56:28.849108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:28.835070    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:29.232290 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:29.232262    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"]
Oct 29 22:56:29.232973 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:56:29.232945    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddddcb392_8e28_4cce_984c_1f06108f927a.slice/crio-1fc4935eaa0d2e8098414d4bcc8ed7a4ab4a0e4b1137c0e496cfc75b71f97370 WatchSource:0}: Error finding container 1fc4935eaa0d2e8098414d4bcc8ed7a4ab4a0e4b1137c0e496cfc75b71f97370: Status 404 returned error can't find the container with id 1fc4935eaa0d2e8098414d4bcc8ed7a4ab4a0e4b1137c0e496cfc75b71f97370
Oct 29 22:56:29.294798 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:29.294763    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" event={"ID":"dddcb392-8e28-4cce-984c-1f06108f927a","Type":"ContainerStarted","Data":"1fc4935eaa0d2e8098414d4bcc8ed7a4ab4a0e4b1137c0e496cfc75b71f97370"}
Oct 29 22:56:29.743408 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:29.743368    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:56:29.743628 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:29.743429    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:56:30.326864 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:30.326823    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" event={"ID":"dddcb392-8e28-4cce-984c-1f06108f927a","Type":"ContainerStarted","Data":"7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6"}
Oct 29 22:56:30.743993 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:30.743963    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:56:30.744238 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:30.744215    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:56:30.836447 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:30.836417    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:31.743756 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:31.743717    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:56:31.744185 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:31.743789    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:56:32.743770 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:32.743732    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:56:32.744237 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:32.743784    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:56:33.743949 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:33.743909    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 22:56:33.744371 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:33.743973    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 22:56:34.749014 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:34.748977    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:34.749489 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:34.749470    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:34.750249 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:34.750226    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:56:34.803866 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:34.803808    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podStartSLOduration=6.803795634 podStartE2EDuration="6.803795634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:30.403438432 +0000 UTC m=+847.544180017" watchObservedRunningTime="2026-10-29 22:56:34.803795634 +0000 UTC m=+851.944537247"
Oct 29 22:56:35.681686 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.681604    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"]
Oct 29 22:56:35.686181 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.686158    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.712391 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.712364    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:35.712617 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.712502    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:56:35.718003 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.717977    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"]
Oct 29 22:56:35.789516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.789489    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.789873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.789569    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qvm\" (UniqueName: \"kubernetes.io/projected/d9da0961-572b-49d6-80b4-76eb7e7aba38-kube-api-access-d7qvm\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.789873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.789731    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.891194 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.891158    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qvm\" (UniqueName: \"kubernetes.io/projected/d9da0961-572b-49d6-80b4-76eb7e7aba38-kube-api-access-d7qvm\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.891379 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.891273    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.891379 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.891374    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.891977 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.891957    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.893629 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.893610    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:35.909948 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:35.909923    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qvm\" (UniqueName: \"kubernetes.io/projected/d9da0961-572b-49d6-80b4-76eb7e7aba38-kube-api-access-d7qvm\") pod \"maas-api-revoke-sub-bbb90a3b-wb7d7\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:36.044215 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:36.044131    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:36.341602 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:36.341573    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"]
Oct 29 22:56:36.342473 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:56:36.342444    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9da0961_572b_49d6_80b4_76eb7e7aba38.slice/crio-05cc480be1d353215f5fbdf70687fb5b446c1d6ec56747d80951787286cc3ead WatchSource:0}: Error finding container 05cc480be1d353215f5fbdf70687fb5b446c1d6ec56747d80951787286cc3ead: Status 404 returned error can't find the container with id 05cc480be1d353215f5fbdf70687fb5b446c1d6ec56747d80951787286cc3ead
Oct 29 22:56:36.365735 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:36.365708    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7" event={"ID":"d9da0961-572b-49d6-80b4-76eb7e7aba38","Type":"ContainerStarted","Data":"05cc480be1d353215f5fbdf70687fb5b446c1d6ec56747d80951787286cc3ead"}
Oct 29 22:56:37.373533 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:37.373492    4742 generic.go:356] "Generic (PLEG): container finished" podID="d9da0961-572b-49d6-80b4-76eb7e7aba38" containerID="01887eaa789371afef5a04055545f07818e9d15525f7cb796dd3a13e13bc8988" exitCode=0
Oct 29 22:56:37.373533 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:37.373528    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7" event={"ID":"d9da0961-572b-49d6-80b4-76eb7e7aba38","Type":"ContainerDied","Data":"01887eaa789371afef5a04055545f07818e9d15525f7cb796dd3a13e13bc8988"}
Oct 29 22:56:38.537815 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.537791    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:38.627827 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.627797    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9da0961-572b-49d6-80b4-76eb7e7aba38-kube-api-access-d7qvm\" (UniqueName: \"kubernetes.io/projected/d9da0961-572b-49d6-80b4-76eb7e7aba38-kube-api-access-d7qvm\") pod \"d9da0961-572b-49d6-80b4-76eb7e7aba38\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") "
Oct 29 22:56:38.628038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.627874    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-openshift-ca\") pod \"d9da0961-572b-49d6-80b4-76eb7e7aba38\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") "
Oct 29 22:56:38.628038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.628001    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-xks-ca\") pod \"d9da0961-572b-49d6-80b4-76eb7e7aba38\" (UID: \"d9da0961-572b-49d6-80b4-76eb7e7aba38\") "
Oct 29 22:56:38.628335 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.628297    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-openshift-ca" pod "d9da0961-572b-49d6-80b4-76eb7e7aba38" (UID: "d9da0961-572b-49d6-80b4-76eb7e7aba38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:38.628582 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.628558    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:38.630184 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.630125    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9da0961-572b-49d6-80b4-76eb7e7aba38-kube-api-access-d7qvm" pod "d9da0961-572b-49d6-80b4-76eb7e7aba38" (UID: "d9da0961-572b-49d6-80b4-76eb7e7aba38"). InnerVolumeSpecName "kube-api-access-d7qvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:38.630255 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.630182    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-xks-ca" pod "d9da0961-572b-49d6-80b4-76eb7e7aba38" (UID: "d9da0961-572b-49d6-80b4-76eb7e7aba38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:38.729503 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.729465    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9da0961-572b-49d6-80b4-76eb7e7aba38-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:38.729503 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.729497    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7qvm\" (UniqueName: \"kubernetes.io/projected/d9da0961-572b-49d6-80b4-76eb7e7aba38-kube-api-access-d7qvm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:38.835966 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.835930    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:38.848886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:38.848858    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:39.387323 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:39.387293    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"
Oct 29 22:56:39.387475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:39.387321    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7" event={"ID":"d9da0961-572b-49d6-80b4-76eb7e7aba38","Type":"ContainerDied","Data":"05cc480be1d353215f5fbdf70687fb5b446c1d6ec56747d80951787286cc3ead"}
Oct 29 22:56:39.387475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:39.387373    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05cc480be1d353215f5fbdf70687fb5b446c1d6ec56747d80951787286cc3ead"
Oct 29 22:56:39.399014 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:39.398990    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:56:48.196638 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.196582    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"]
Oct 29 22:56:48.197307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.197133    4742 cpu_manager.go:414] "containerMap: removing container" podUID="d9da0961-572b-49d6-80b4-76eb7e7aba38" containerName="revoke-keys"
Oct 29 22:56:48.197307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.197146    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9da0961-572b-49d6-80b4-76eb7e7aba38" containerName="revoke-keys"
Oct 29 22:56:48.197307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.197272    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9da0961-572b-49d6-80b4-76eb7e7aba38" containerName="revoke-keys"
Oct 29 22:56:48.200859 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.200835    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.221837 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.221809    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"]
Oct 29 22:56:48.229595 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.227747    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:48.229595 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.227778    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:56:48.360757 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.360720    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.360989 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.360786    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb58k\" (UniqueName: \"kubernetes.io/projected/c8d033e3-b179-46fd-9d28-25ebf951f150-kube-api-access-bb58k\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.360989 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.360952    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.462063 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.461977    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.462241 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.462062    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb58k\" (UniqueName: \"kubernetes.io/projected/c8d033e3-b179-46fd-9d28-25ebf951f150-kube-api-access-bb58k\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.462241 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.462191    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.462640 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.462620    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.464731 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.464706    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.482439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.482409    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb58k\" (UniqueName: \"kubernetes.io/projected/c8d033e3-b179-46fd-9d28-25ebf951f150-kube-api-access-bb58k\") pod \"maas-api-revoke-sub-bc849176-7b2cg\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.514715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.514682    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:48.831718 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:48.831684    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"]
Oct 29 22:56:49.454947 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:49.454840    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg" event={"ID":"c8d033e3-b179-46fd-9d28-25ebf951f150","Type":"ContainerStarted","Data":"639ff5ec8e83f6dcd97fcd0e74ac4d7b3f74a9f78f2ea845335ac312509bbfed"}
Oct 29 22:56:49.454947 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:49.454906    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg" event={"ID":"c8d033e3-b179-46fd-9d28-25ebf951f150","Type":"ContainerStarted","Data":"ca1c393c038ee4cc9b4ed65cfb02f8e8bd0ea3e6c641a358354fa8707a26c1b8"}
Oct 29 22:56:49.495501 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:49.495425    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg" podStartSLOduration=1.495406409 podStartE2EDuration="1.495406409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:49.49211231 +0000 UTC m=+866.632853895" watchObservedRunningTime="2026-10-29 22:56:49.495406409 +0000 UTC m=+866.636147996"
Oct 29 22:56:50.501109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:50.500730    4742 generic.go:356] "Generic (PLEG): container finished" podID="c8d033e3-b179-46fd-9d28-25ebf951f150" containerID="639ff5ec8e83f6dcd97fcd0e74ac4d7b3f74a9f78f2ea845335ac312509bbfed" exitCode=0
Oct 29 22:56:50.501109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:50.500911    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg" event={"ID":"c8d033e3-b179-46fd-9d28-25ebf951f150","Type":"ContainerDied","Data":"639ff5ec8e83f6dcd97fcd0e74ac4d7b3f74a9f78f2ea845335ac312509bbfed"}
Oct 29 22:56:51.329110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:51.316197    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.396110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:51.393226    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.483108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:51.482181    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.575110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:51.574178    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.606863 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:51.606815    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.607278 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:51.607140    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.750210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:51.750161    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:56:51.811192 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:51.811151    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.231210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.227655    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"]
Oct 29 22:56:52.235678 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.233999    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"]
Oct 29 22:56:52.235678 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.234133    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.247385 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.247038    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:52.260116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.260064    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.260552 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.260528    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkp9\" (UniqueName: \"kubernetes.io/projected/54fa6f22-c750-43c0-ad37-49fe44224899-kube-api-access-bgkp9\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.260737 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.260719    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.361585 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.361549    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-openshift-ca\") pod \"c8d033e3-b179-46fd-9d28-25ebf951f150\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") "
Oct 29 22:56:52.361764 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.361689    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-xks-ca\") pod \"c8d033e3-b179-46fd-9d28-25ebf951f150\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") "
Oct 29 22:56:52.361807 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.361773    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d033e3-b179-46fd-9d28-25ebf951f150-kube-api-access-bb58k\" (UniqueName: \"kubernetes.io/projected/c8d033e3-b179-46fd-9d28-25ebf951f150-kube-api-access-bb58k\") pod \"c8d033e3-b179-46fd-9d28-25ebf951f150\" (UID: \"c8d033e3-b179-46fd-9d28-25ebf951f150\") "
Oct 29 22:56:52.362010 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.361977    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-openshift-ca" pod "c8d033e3-b179-46fd-9d28-25ebf951f150" (UID: "c8d033e3-b179-46fd-9d28-25ebf951f150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:52.362128 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.361986    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.362243 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.362187    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkp9\" (UniqueName: \"kubernetes.io/projected/54fa6f22-c750-43c0-ad37-49fe44224899-kube-api-access-bgkp9\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.362315 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.362245    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.362533 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.362509    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:52.363383 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.363355    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.364774 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.364740    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d033e3-b179-46fd-9d28-25ebf951f150-kube-api-access-bb58k" pod "c8d033e3-b179-46fd-9d28-25ebf951f150" (UID: "c8d033e3-b179-46fd-9d28-25ebf951f150"). InnerVolumeSpecName "kube-api-access-bb58k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:52.364982 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.364956    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-xks-ca" pod "c8d033e3-b179-46fd-9d28-25ebf951f150" (UID: "c8d033e3-b179-46fd-9d28-25ebf951f150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:52.365318 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.365295    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.380833 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.380804    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkp9\" (UniqueName: \"kubernetes.io/projected/54fa6f22-c750-43c0-ad37-49fe44224899-kube-api-access-bgkp9\") pod \"maas-api-revoke-sub-1b76cef9-w8zjs\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.463448 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.463407    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb58k\" (UniqueName: \"kubernetes.io/projected/c8d033e3-b179-46fd-9d28-25ebf951f150-kube-api-access-bb58k\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:52.463448 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.463440    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d033e3-b179-46fd-9d28-25ebf951f150-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:52.531785 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.531755    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"
Oct 29 22:56:52.531952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.531789    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg" event={"ID":"c8d033e3-b179-46fd-9d28-25ebf951f150","Type":"ContainerDied","Data":"ca1c393c038ee4cc9b4ed65cfb02f8e8bd0ea3e6c641a358354fa8707a26c1b8"}
Oct 29 22:56:52.531952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.531840    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1c393c038ee4cc9b4ed65cfb02f8e8bd0ea3e6c641a358354fa8707a26c1b8"
Oct 29 22:56:52.582571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.582537    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:52.889034 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:52.888997    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"]
Oct 29 22:56:52.890178 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:56:52.890148    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fa6f22_c750_43c0_ad37_49fe44224899.slice/crio-446ed0418dfa61e9e64d3694d55857a38a29769c71b19af45d74eaaaf05b2870 WatchSource:0}: Error finding container 446ed0418dfa61e9e64d3694d55857a38a29769c71b19af45d74eaaaf05b2870: Status 404 returned error can't find the container with id 446ed0418dfa61e9e64d3694d55857a38a29769c71b19af45d74eaaaf05b2870
Oct 29 22:56:53.129693 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:56:53.129652    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:56:53.545150 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:53.544119    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs" event={"ID":"54fa6f22-c750-43c0-ad37-49fe44224899","Type":"ContainerStarted","Data":"96370aaaf20b8052a4abbbea07f8fd79595aac7fe7f31615f47c02c0013df938"}
Oct 29 22:56:53.545150 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:53.544193    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs" event={"ID":"54fa6f22-c750-43c0-ad37-49fe44224899","Type":"ContainerStarted","Data":"446ed0418dfa61e9e64d3694d55857a38a29769c71b19af45d74eaaaf05b2870"}
Oct 29 22:56:53.581894 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:53.581759    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs" podStartSLOduration=1.581744188 podStartE2EDuration="1.581744188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:53.580911428 +0000 UTC m=+870.721653009" watchObservedRunningTime="2026-10-29 22:56:53.581744188 +0000 UTC m=+870.722485773"
Oct 29 22:56:54.574164 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:54.574116    4742 generic.go:356] "Generic (PLEG): container finished" podID="54fa6f22-c750-43c0-ad37-49fe44224899" containerID="96370aaaf20b8052a4abbbea07f8fd79595aac7fe7f31615f47c02c0013df938" exitCode=0
Oct 29 22:56:54.574631 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:54.574176    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs" event={"ID":"54fa6f22-c750-43c0-ad37-49fe44224899","Type":"ContainerDied","Data":"96370aaaf20b8052a4abbbea07f8fd79595aac7fe7f31615f47c02c0013df938"}
Oct 29 22:56:55.731782 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.731757    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:55.809782 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.809749    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-openshift-ca\") pod \"54fa6f22-c750-43c0-ad37-49fe44224899\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") "
Oct 29 22:56:55.809977 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.809843    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fa6f22-c750-43c0-ad37-49fe44224899-kube-api-access-bgkp9\" (UniqueName: \"kubernetes.io/projected/54fa6f22-c750-43c0-ad37-49fe44224899-kube-api-access-bgkp9\") pod \"54fa6f22-c750-43c0-ad37-49fe44224899\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") "
Oct 29 22:56:55.809977 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.809955    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-xks-ca\") pod \"54fa6f22-c750-43c0-ad37-49fe44224899\" (UID: \"54fa6f22-c750-43c0-ad37-49fe44224899\") "
Oct 29 22:56:55.810236 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.810204    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-openshift-ca" pod "54fa6f22-c750-43c0-ad37-49fe44224899" (UID: "54fa6f22-c750-43c0-ad37-49fe44224899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:55.810506 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.810469    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:55.812540 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.812505    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fa6f22-c750-43c0-ad37-49fe44224899-kube-api-access-bgkp9" pod "54fa6f22-c750-43c0-ad37-49fe44224899" (UID: "54fa6f22-c750-43c0-ad37-49fe44224899"). InnerVolumeSpecName "kube-api-access-bgkp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:55.812648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.812571    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-xks-ca" pod "54fa6f22-c750-43c0-ad37-49fe44224899" (UID: "54fa6f22-c750-43c0-ad37-49fe44224899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:55.911677 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.911643    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fa6f22-c750-43c0-ad37-49fe44224899-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:55.911677 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:55.911672    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgkp9\" (UniqueName: \"kubernetes.io/projected/54fa6f22-c750-43c0-ad37-49fe44224899-kube-api-access-bgkp9\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:56:56.184010 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.183919    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"]
Oct 29 22:56:56.184692 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.184664    4742 cpu_manager.go:414] "containerMap: removing container" podUID="54fa6f22-c750-43c0-ad37-49fe44224899" containerName="revoke-keys"
Oct 29 22:56:56.184692 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.184690    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fa6f22-c750-43c0-ad37-49fe44224899" containerName="revoke-keys"
Oct 29 22:56:56.184858 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.184711    4742 cpu_manager.go:414] "containerMap: removing container" podUID="c8d033e3-b179-46fd-9d28-25ebf951f150" containerName="revoke-keys"
Oct 29 22:56:56.184858 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.184718    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d033e3-b179-46fd-9d28-25ebf951f150" containerName="revoke-keys"
Oct 29 22:56:56.184965 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.184901    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fa6f22-c750-43c0-ad37-49fe44224899" containerName="revoke-keys"
Oct 29 22:56:56.184965 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.184915    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d033e3-b179-46fd-9d28-25ebf951f150" containerName="revoke-keys"
Oct 29 22:56:56.188562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.188539    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216308 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216277    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216462 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216408    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5md\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-kube-api-access-xg5md\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216519 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216496    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216563 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216534    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26438b13-81c6-4c88-a88e-19bceef48774-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216708    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216775 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216750    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216825 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216800    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216875 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216834    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.216875 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.216862    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26438b13-81c6-4c88-a88e-19bceef48774-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.223824 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.223795    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"]
Oct 29 22:56:56.317871 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.317813    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.317871 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.317874    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318149 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.317901    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318149 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.317916    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318149 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.317934    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26438b13-81c6-4c88-a88e-19bceef48774-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318149 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.317987    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318149 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.318143    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5md\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-kube-api-access-xg5md\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318428 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.318201    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318428 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.318220    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26438b13-81c6-4c88-a88e-19bceef48774-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318428 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.318294    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318428 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.318334    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318750 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.318726    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.318824 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.318751    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.319223 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.319204    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26438b13-81c6-4c88-a88e-19bceef48774-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.320293 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.320269    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.321036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.321016    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26438b13-81c6-4c88-a88e-19bceef48774-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.336996 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.336968    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.337120 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.337057    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5md\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-kube-api-access-xg5md\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.501938 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.501858    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:56.588775 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.588745    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"
Oct 29 22:56:56.588775 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.588763    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs" event={"ID":"54fa6f22-c750-43c0-ad37-49fe44224899","Type":"ContainerDied","Data":"446ed0418dfa61e9e64d3694d55857a38a29769c71b19af45d74eaaaf05b2870"}
Oct 29 22:56:56.589013 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.588841    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="446ed0418dfa61e9e64d3694d55857a38a29769c71b19af45d74eaaaf05b2870"
Oct 29 22:56:56.823530 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.823501    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"]
Oct 29 22:56:56.823933 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:56:56.823905    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26438b13_81c6_4c88_a88e_19bceef48774.slice/crio-b854856c108525168be2eca99a0af6275a5404108d49e853c7cbec1dd42703bf WatchSource:0}: Error finding container b854856c108525168be2eca99a0af6275a5404108d49e853c7cbec1dd42703bf: Status 404 returned error can't find the container with id b854856c108525168be2eca99a0af6275a5404108d49e853c7cbec1dd42703bf
Oct 29 22:56:56.826704 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.826661    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:56:56.826855 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.826752    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:56:56.826855 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:56.826798    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:56:57.598136 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:57.598072    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" event={"ID":"26438b13-81c6-4c88-a88e-19bceef48774","Type":"ContainerStarted","Data":"db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36"}
Oct 29 22:56:57.598331 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:57.598185    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" event={"ID":"26438b13-81c6-4c88-a88e-19bceef48774","Type":"ContainerStarted","Data":"b854856c108525168be2eca99a0af6275a5404108d49e853c7cbec1dd42703bf"}
Oct 29 22:56:57.673875 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:57.673789    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" podStartSLOduration=1.673764468 podStartE2EDuration="1.673764468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:57.649777953 +0000 UTC m=+874.790519538" watchObservedRunningTime="2026-10-29 22:56:57.673764468 +0000 UTC m=+874.814506108"
Oct 29 22:56:57.992212 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:57.992118    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"]
Oct 29 22:56:57.996365 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:57.996344    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.012001 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.011970    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:58.012725 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.012702    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:56:58.015715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.015688    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"]
Oct 29 22:56:58.041145 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.041105    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.041388 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.041268    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqjj\" (UniqueName: \"kubernetes.io/projected/50163722-c649-46d9-911a-85ec7832e7f5-kube-api-access-rzqjj\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.041388 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.041334    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.142952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.142905    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.143195 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.143010    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqjj\" (UniqueName: \"kubernetes.io/projected/50163722-c649-46d9-911a-85ec7832e7f5-kube-api-access-rzqjj\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.143195 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.143053    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.143758 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.143723    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.145259 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.145239    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.184578 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.184542    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqjj\" (UniqueName: \"kubernetes.io/projected/50163722-c649-46d9-911a-85ec7832e7f5-kube-api-access-rzqjj\") pod \"maas-api-revoke-sub-1d86588b-q7mng\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.309942 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.309849    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:56:58.504508 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.502897    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:56:58.504508 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.504440    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:56:58.504717 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.504489    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 29 22:56:58.685580 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:58.685551    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"]
Oct 29 22:56:58.688196 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:56:58.688161    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50163722_c649_46d9_911a_85ec7832e7f5.slice/crio-8898d1a24c2684869eb510415ddfd863f2d3fce95fdfefaee3e40ba4c04c305c WatchSource:0}: Error finding container 8898d1a24c2684869eb510415ddfd863f2d3fce95fdfefaee3e40ba4c04c305c: Status 404 returned error can't find the container with id 8898d1a24c2684869eb510415ddfd863f2d3fce95fdfefaee3e40ba4c04c305c
Oct 29 22:56:59.502518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:59.502437    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:56:59.502518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:59.502488    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 29 22:56:59.615136 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:59.615099    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng" event={"ID":"50163722-c649-46d9-911a-85ec7832e7f5","Type":"ContainerStarted","Data":"d9a4f369202bd537f6ba03b51e5e13af306d4cba38e0bcd95ef6cfd8119e7fc7"}
Oct 29 22:56:59.615293 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:56:59.615157    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng" event={"ID":"50163722-c649-46d9-911a-85ec7832e7f5","Type":"ContainerStarted","Data":"8898d1a24c2684869eb510415ddfd863f2d3fce95fdfefaee3e40ba4c04c305c"}
Oct 29 22:57:00.502475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:00.502438    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:57:00.502656 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:00.502491    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 29 22:57:00.622838 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:00.622802    4742 generic.go:356] "Generic (PLEG): container finished" podID="50163722-c649-46d9-911a-85ec7832e7f5" containerID="d9a4f369202bd537f6ba03b51e5e13af306d4cba38e0bcd95ef6cfd8119e7fc7" exitCode=0
Oct 29 22:57:00.623036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:00.622884    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng" event={"ID":"50163722-c649-46d9-911a-85ec7832e7f5","Type":"ContainerDied","Data":"d9a4f369202bd537f6ba03b51e5e13af306d4cba38e0bcd95ef6cfd8119e7fc7"}
Oct 29 22:57:01.503062 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.503024    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:57:01.503457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.503097    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 29 22:57:01.839516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.839479    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:57:01.913689 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.913657    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-openshift-ca\") pod \"50163722-c649-46d9-911a-85ec7832e7f5\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") "
Oct 29 22:57:01.913845 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.913778    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50163722-c649-46d9-911a-85ec7832e7f5-kube-api-access-rzqjj\" (UniqueName: \"kubernetes.io/projected/50163722-c649-46d9-911a-85ec7832e7f5-kube-api-access-rzqjj\") pod \"50163722-c649-46d9-911a-85ec7832e7f5\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") "
Oct 29 22:57:01.913930 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.913898    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-xks-ca\") pod \"50163722-c649-46d9-911a-85ec7832e7f5\" (UID: \"50163722-c649-46d9-911a-85ec7832e7f5\") "
Oct 29 22:57:01.914123 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.914068    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-openshift-ca" pod "50163722-c649-46d9-911a-85ec7832e7f5" (UID: "50163722-c649-46d9-911a-85ec7832e7f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:01.914620 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.914602    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:01.916610 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.916578    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50163722-c649-46d9-911a-85ec7832e7f5-kube-api-access-rzqjj" pod "50163722-c649-46d9-911a-85ec7832e7f5" (UID: "50163722-c649-46d9-911a-85ec7832e7f5"). InnerVolumeSpecName "kube-api-access-rzqjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:01.918299 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:01.918264    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-xks-ca" pod "50163722-c649-46d9-911a-85ec7832e7f5" (UID: "50163722-c649-46d9-911a-85ec7832e7f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:02.016112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.016046    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzqjj\" (UniqueName: \"kubernetes.io/projected/50163722-c649-46d9-911a-85ec7832e7f5-kube-api-access-rzqjj\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:02.016112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.016111    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50163722-c649-46d9-911a-85ec7832e7f5-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:02.502315 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.502274    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:57:02.502534 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.502339    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 29 22:57:02.637056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.637023    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"
Oct 29 22:57:02.637487 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.637026    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng" event={"ID":"50163722-c649-46d9-911a-85ec7832e7f5","Type":"ContainerDied","Data":"8898d1a24c2684869eb510415ddfd863f2d3fce95fdfefaee3e40ba4c04c305c"}
Oct 29 22:57:02.637487 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.637173    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8898d1a24c2684869eb510415ddfd863f2d3fce95fdfefaee3e40ba4c04c305c"
Oct 29 22:57:02.710207 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.710177    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"]
Oct 29 22:57:02.710731 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.710717    4742 cpu_manager.go:414] "containerMap: removing container" podUID="50163722-c649-46d9-911a-85ec7832e7f5" containerName="revoke-keys"
Oct 29 22:57:02.710775 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.710733    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50163722-c649-46d9-911a-85ec7832e7f5" containerName="revoke-keys"
Oct 29 22:57:02.710847 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.710838    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="50163722-c649-46d9-911a-85ec7832e7f5" containerName="revoke-keys"
Oct 29 22:57:02.714439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.714417    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.729770 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.729717    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:02.729770 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.729768    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:02.745118 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.745045    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"]
Oct 29 22:57:02.829924 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.829832    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.829924 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.829908    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.830143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.830073    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7v4\" (UniqueName: \"kubernetes.io/projected/a827df17-86d8-4625-884f-d9eff3cdccdc-kube-api-access-nb7v4\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.931169 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.931128    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.931373 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.931240    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.931373 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.931317    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7v4\" (UniqueName: \"kubernetes.io/projected/a827df17-86d8-4625-884f-d9eff3cdccdc-kube-api-access-nb7v4\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.931893 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.931869    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.933651 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.933629    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:02.951135 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:02.951106    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7v4\" (UniqueName: \"kubernetes.io/projected/a827df17-86d8-4625-884f-d9eff3cdccdc-kube-api-access-nb7v4\") pod \"maas-api-revoke-sub-f25aadaf-p8cb8\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:03.030551 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:03.030512    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:03.341098 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:03.341031    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"]
Oct 29 22:57:03.342460 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:03.342428    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda827df17_86d8_4625_884f_d9eff3cdccdc.slice/crio-fd4c21509e54842f1673ba449abcd4d54ac32368f917a0b74c66b42c577e4ca0 WatchSource:0}: Error finding container fd4c21509e54842f1673ba449abcd4d54ac32368f917a0b74c66b42c577e4ca0: Status 404 returned error can't find the container with id fd4c21509e54842f1673ba449abcd4d54ac32368f917a0b74c66b42c577e4ca0
Oct 29 22:57:03.503205 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:03.503173    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:57:03.503418 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:03.503249    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 29 22:57:03.644763 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:03.644680    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8" event={"ID":"a827df17-86d8-4625-884f-d9eff3cdccdc","Type":"ContainerStarted","Data":"fd4c21509e54842f1673ba449abcd4d54ac32368f917a0b74c66b42c577e4ca0"}
Oct 29 22:57:04.512052 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:04.511059    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:57:04.513003 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:04.512957    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:57:04.515001 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:04.514932    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 22:57:04.654652 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:04.654612    4742 generic.go:356] "Generic (PLEG): container finished" podID="a827df17-86d8-4625-884f-d9eff3cdccdc" containerID="d68a11f7a71763e5cd08207cd6c4012303d140a813075e904c1c53f8feaa8f5a" exitCode=0
Oct 29 22:57:04.655207 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:04.654893    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8" event={"ID":"a827df17-86d8-4625-884f-d9eff3cdccdc","Type":"ContainerDied","Data":"d68a11f7a71763e5cd08207cd6c4012303d140a813075e904c1c53f8feaa8f5a"}
Oct 29 22:57:06.229110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.228271    4742 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:06.229110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.228342    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.229110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.228829    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.318111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.306156    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:57:06.406116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.401185    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.561308 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.502183    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.728309 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.577262    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.728309 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.691753    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.728309 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.701039    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.728309 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.701394    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:57:06.759197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.757895    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:06.865580 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.839171    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.869060 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.867800    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.992636 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:06.992265    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:07.052107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.051509    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.134175 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.130512    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-openshift-ca\") pod \"a827df17-86d8-4625-884f-d9eff3cdccdc\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") "
Oct 29 22:57:07.150613 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.150561    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a827df17-86d8-4625-884f-d9eff3cdccdc-kube-api-access-nb7v4\" (UniqueName: \"kubernetes.io/projected/a827df17-86d8-4625-884f-d9eff3cdccdc-kube-api-access-nb7v4\") pod \"a827df17-86d8-4625-884f-d9eff3cdccdc\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") "
Oct 29 22:57:07.151000 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.150972    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-xks-ca\") pod \"a827df17-86d8-4625-884f-d9eff3cdccdc\" (UID: \"a827df17-86d8-4625-884f-d9eff3cdccdc\") "
Oct 29 22:57:07.177775 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.177734    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-openshift-ca" pod "a827df17-86d8-4625-884f-d9eff3cdccdc" (UID: "a827df17-86d8-4625-884f-d9eff3cdccdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:07.206913 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.206872    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-xks-ca" pod "a827df17-86d8-4625-884f-d9eff3cdccdc" (UID: "a827df17-86d8-4625-884f-d9eff3cdccdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:07.224487 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.224449    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded"
Oct 29 22:57:07.225331 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.225298    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a827df17-86d8-4625-884f-d9eff3cdccdc-kube-api-access-nb7v4" pod "a827df17-86d8-4625-884f-d9eff3cdccdc" (UID: "a827df17-86d8-4625-884f-d9eff3cdccdc"). InnerVolumeSpecName "kube-api-access-nb7v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:07.233021 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.232984    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.247129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.238569    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:57:07.299124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.274121    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7v4\" (UniqueName: \"kubernetes.io/projected/a827df17-86d8-4625-884f-d9eff3cdccdc-kube-api-access-nb7v4\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:07.299124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.274156    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:07.299124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.274171    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a827df17-86d8-4625-884f-d9eff3cdccdc-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:07.330289 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.330210    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.331189 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.331061    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.359925 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.359199    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.361102 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.360236    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:57:07.405168 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.404222    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.410108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.409781    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:57:07.548944 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.548900    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.557015 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.556982    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:57:07.607124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.607062    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.636117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.627203    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:57:07.645506 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.645478    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:57:07.705354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.697635    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:07.705354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.698750    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:57:07.705354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.698807    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:57:07.705354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.704175    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.705354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.704413    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.705354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.704604    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:57:07.705354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.704654    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:57:07.759621 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.759510    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:07.779058 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.779030    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:57:07.833116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.833078    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:57:07.861907 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.861881    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:57:07.885392 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.885343    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.886051 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.886025    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:57:07.886925 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.886894    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.887543 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.887519    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:57:07.894564 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.894539    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:57:07.898519 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.898497    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:57:07.898833 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.898819    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:57:07.925750 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.925722    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:57:07.925973 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.925960    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:57:07.926113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.926102    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:57:07.926229 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.926219    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:57:07.938565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.935680    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"
Oct 29 22:57:07.938565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.937145    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8" event={"ID":"a827df17-86d8-4625-884f-d9eff3cdccdc","Type":"ContainerDied","Data":"fd4c21509e54842f1673ba449abcd4d54ac32368f917a0b74c66b42c577e4ca0"}
Oct 29 22:57:07.938565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:07.937189    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd4c21509e54842f1673ba449abcd4d54ac32368f917a0b74c66b42c577e4ca0"
Oct 29 22:57:09.864492 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:09.864459    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"]
Oct 29 22:57:09.865110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:09.865072    4742 cpu_manager.go:414] "containerMap: removing container" podUID="a827df17-86d8-4625-884f-d9eff3cdccdc" containerName="revoke-keys"
Oct 29 22:57:09.865110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:09.865101    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a827df17-86d8-4625-884f-d9eff3cdccdc" containerName="revoke-keys"
Oct 29 22:57:09.865216 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:09.865206    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="a827df17-86d8-4625-884f-d9eff3cdccdc" containerName="revoke-keys"
Oct 29 22:57:09.868694 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:09.868668    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:09.887342 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:09.887316    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"]
Oct 29 22:57:09.894598 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:09.894572    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:09.894738 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:09.894643    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:10.015126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.015065    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.015297 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.015138    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444fg\" (UniqueName: \"kubernetes.io/projected/c66fb512-f4b1-40dd-9650-d1c20240cbc4-kube-api-access-444fg\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.015421 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.015395    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.116844 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.116750    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.116844 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.116809    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.117036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.116847    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444fg\" (UniqueName: \"kubernetes.io/projected/c66fb512-f4b1-40dd-9650-d1c20240cbc4-kube-api-access-444fg\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.117567 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.117544    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.119220 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.119195    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.135902 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.135878    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-444fg\" (UniqueName: \"kubernetes.io/projected/c66fb512-f4b1-40dd-9650-d1c20240cbc4-kube-api-access-444fg\") pod \"maas-api-revoke-sub-975010f8-qfprf\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.182052 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.182013    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:10.485461 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.485436    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"]
Oct 29 22:57:10.486061 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:10.486037    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66fb512_f4b1_40dd_9650_d1c20240cbc4.slice/crio-7ffa9608551079ebbe34399cbb906a6a6f65cb1e7a64b4e10acc11cb8177b955 WatchSource:0}: Error finding container 7ffa9608551079ebbe34399cbb906a6a6f65cb1e7a64b4e10acc11cb8177b955: Status 404 returned error can't find the container with id 7ffa9608551079ebbe34399cbb906a6a6f65cb1e7a64b4e10acc11cb8177b955
Oct 29 22:57:10.831261 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.831224    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"]
Oct 29 22:57:10.836383 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.836358    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.913620 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.913583    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"]
Oct 29 22:57:10.927659 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.927617    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692fbafb-57c8-4794-a6c4-c159075ae618-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.927919 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.927890    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.928122 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.927941    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.928122 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.927999    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.928122 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.928052    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.928324 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.928166    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.928324 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.928196    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.928324 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.928287    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692fbafb-57c8-4794-a6c4-c159075ae618-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.928534 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.928334    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfwn\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-kube-api-access-cwfwn\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:10.957526 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.957485    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf" event={"ID":"c66fb512-f4b1-40dd-9650-d1c20240cbc4","Type":"ContainerStarted","Data":"1ec94ddec8eeebe2707ac9d2d10f9b8cb6254a94c203d32a8b37eea4783d26df"}
Oct 29 22:57:10.957724 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:10.957550    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf" event={"ID":"c66fb512-f4b1-40dd-9650-d1c20240cbc4","Type":"ContainerStarted","Data":"7ffa9608551079ebbe34399cbb906a6a6f65cb1e7a64b4e10acc11cb8177b955"}
Oct 29 22:57:11.012777 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.012721    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf" podStartSLOduration=2.012707414 podStartE2EDuration="2.012707414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:11.009310775 +0000 UTC m=+888.150052356" watchObservedRunningTime="2026-10-29 22:57:11.012707414 +0000 UTC m=+888.153449015"
Oct 29 22:57:11.030143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030048    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030104    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030140    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030173    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030257    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030282    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030363    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692fbafb-57c8-4794-a6c4-c159075ae618-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030611 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030408    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfwn\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-kube-api-access-cwfwn\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030611 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030497    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692fbafb-57c8-4794-a6c4-c159075ae618-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030611 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030581    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030610    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030685    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.030749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.030712    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.031429 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.031401    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692fbafb-57c8-4794-a6c4-c159075ae618-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.032579 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.032556    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.032729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.032711    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692fbafb-57c8-4794-a6c4-c159075ae618-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.054067 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.054035    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.054235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.054217    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfwn\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-kube-api-access-cwfwn\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.153278 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.153240    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:11.479830 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.477877    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"]
Oct 29 22:57:11.480017 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:11.479925    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692fbafb_57c8_4794_a6c4_c159075ae618.slice/crio-1ec12a69c81a587ad6080ff6b11cd629eaa4cd3b91ddffe12b99d2c8cf6105d5 WatchSource:0}: Error finding container 1ec12a69c81a587ad6080ff6b11cd629eaa4cd3b91ddffe12b99d2c8cf6105d5: Status 404 returned error can't find the container with id 1ec12a69c81a587ad6080ff6b11cd629eaa4cd3b91ddffe12b99d2c8cf6105d5
Oct 29 22:57:11.482500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.482462    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:11.482590 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.482558    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:11.482628 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.482602    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:11.965395 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.965359    4742 generic.go:356] "Generic (PLEG): container finished" podID="c66fb512-f4b1-40dd-9650-d1c20240cbc4" containerID="1ec94ddec8eeebe2707ac9d2d10f9b8cb6254a94c203d32a8b37eea4783d26df" exitCode=0
Oct 29 22:57:11.965841 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.965447    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf" event={"ID":"c66fb512-f4b1-40dd-9650-d1c20240cbc4","Type":"ContainerDied","Data":"1ec94ddec8eeebe2707ac9d2d10f9b8cb6254a94c203d32a8b37eea4783d26df"}
Oct 29 22:57:11.967048 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.967018    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" event={"ID":"692fbafb-57c8-4794-a6c4-c159075ae618","Type":"ContainerStarted","Data":"04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611"}
Oct 29 22:57:11.967177 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:11.967069    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" event={"ID":"692fbafb-57c8-4794-a6c4-c159075ae618","Type":"ContainerStarted","Data":"1ec12a69c81a587ad6080ff6b11cd629eaa4cd3b91ddffe12b99d2c8cf6105d5"}
Oct 29 22:57:12.068124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:12.068051    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podStartSLOduration=2.068034066 podStartE2EDuration="2.068034066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:12.063512609 +0000 UTC m=+889.204254192" watchObservedRunningTime="2026-10-29 22:57:12.068034066 +0000 UTC m=+889.208775647"
Oct 29 22:57:12.153941 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:12.153911    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:12.155550 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:12.155521    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:57:12.155665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:12.155576    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:57:13.156896 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.154417    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:57:13.156896 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.154494    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:57:13.193851 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.193819    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:13.292565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.292470    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-xks-ca\") pod \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") "
Oct 29 22:57:13.292565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.292543    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-openshift-ca\") pod \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") "
Oct 29 22:57:13.292795 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.292631    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66fb512-f4b1-40dd-9650-d1c20240cbc4-kube-api-access-444fg\" (UniqueName: \"kubernetes.io/projected/c66fb512-f4b1-40dd-9650-d1c20240cbc4-kube-api-access-444fg\") pod \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\" (UID: \"c66fb512-f4b1-40dd-9650-d1c20240cbc4\") "
Oct 29 22:57:13.292942 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.292913    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-openshift-ca" pod "c66fb512-f4b1-40dd-9650-d1c20240cbc4" (UID: "c66fb512-f4b1-40dd-9650-d1c20240cbc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:13.293373 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.293345    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:13.294981 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.294954    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-xks-ca" pod "c66fb512-f4b1-40dd-9650-d1c20240cbc4" (UID: "c66fb512-f4b1-40dd-9650-d1c20240cbc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:13.295263 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.295237    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66fb512-f4b1-40dd-9650-d1c20240cbc4-kube-api-access-444fg" pod "c66fb512-f4b1-40dd-9650-d1c20240cbc4" (UID: "c66fb512-f4b1-40dd-9650-d1c20240cbc4"). InnerVolumeSpecName "kube-api-access-444fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:13.394147 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.394108    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c66fb512-f4b1-40dd-9650-d1c20240cbc4-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:13.394147 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:13.394145    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-444fg\" (UniqueName: \"kubernetes.io/projected/c66fb512-f4b1-40dd-9650-d1c20240cbc4-kube-api-access-444fg\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:14.005485 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:14.005446    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf" event={"ID":"c66fb512-f4b1-40dd-9650-d1c20240cbc4","Type":"ContainerDied","Data":"7ffa9608551079ebbe34399cbb906a6a6f65cb1e7a64b4e10acc11cb8177b955"}
Oct 29 22:57:14.005485 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:14.005468    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"
Oct 29 22:57:14.005738 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:14.005500    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ffa9608551079ebbe34399cbb906a6a6f65cb1e7a64b4e10acc11cb8177b955"
Oct 29 22:57:14.153928 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:14.153895    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:57:14.154131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:14.153956    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:57:15.153901 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:15.153862    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:57:15.154342 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:15.153920    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:57:16.154494 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.154459    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:57:16.154881 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.154518    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:57:16.284197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.284137    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"]
Oct 29 22:57:16.285049 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.285017    4742 cpu_manager.go:414] "containerMap: removing container" podUID="c66fb512-f4b1-40dd-9650-d1c20240cbc4" containerName="revoke-keys"
Oct 29 22:57:16.285049 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.285045    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66fb512-f4b1-40dd-9650-d1c20240cbc4" containerName="revoke-keys"
Oct 29 22:57:16.286351 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.285269    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66fb512-f4b1-40dd-9650-d1c20240cbc4" containerName="revoke-keys"
Oct 29 22:57:16.289466 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.289445    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.290230 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.290206    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"]
Oct 29 22:57:16.305120 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.304550    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:16.316015 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.315974    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:16.335230 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.335200    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.335375 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.335277    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5l7m\" (UniqueName: \"kubernetes.io/projected/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-kube-api-access-f5l7m\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.335521 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.335481    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.437219 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.437133    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.437385 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.437235    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.437385 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.437294    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5l7m\" (UniqueName: \"kubernetes.io/projected/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-kube-api-access-f5l7m\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.437873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.437846    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.439447 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.439426    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.481899 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.481861    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5l7m\" (UniqueName: \"kubernetes.io/projected/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-kube-api-access-f5l7m\") pod \"maas-api-revoke-sub-179f2906-z5jch\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.614693 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.614659    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:16.948719 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:16.948691    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"]
Oct 29 22:57:16.949735 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:16.949705    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfc5f76_cc2c_4b8c_a60c_73717e53e7b1.slice/crio-ae9171432136fdfdd5322d2464a0aeea3a744be9354014d6d225e03d0d76d0ea WatchSource:0}: Error finding container ae9171432136fdfdd5322d2464a0aeea3a744be9354014d6d225e03d0d76d0ea: Status 404 returned error can't find the container with id ae9171432136fdfdd5322d2464a0aeea3a744be9354014d6d225e03d0d76d0ea
Oct 29 22:57:17.024562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:17.024530    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch" event={"ID":"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1","Type":"ContainerStarted","Data":"ae9171432136fdfdd5322d2464a0aeea3a744be9354014d6d225e03d0d76d0ea"}
Oct 29 22:57:17.155117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:17.155054    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:57:17.155587 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:17.155140    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:57:18.032868 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:18.032836    4742 generic.go:356] "Generic (PLEG): container finished" podID="2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1" containerID="dd4187340b4f7a0450b4f62dab182f717a1b2dbef2a0450ae0468c3c9a4703da" exitCode=0
Oct 29 22:57:18.033007 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:18.032916    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch" event={"ID":"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1","Type":"ContainerDied","Data":"dd4187340b4f7a0450b4f62dab182f717a1b2dbef2a0450ae0468c3c9a4703da"}
Oct 29 22:57:18.153993 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:18.153957    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:57:18.154162 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:18.154017    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 29 22:57:19.159937 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.159904    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:19.160472 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.160346    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:19.161876 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.161851    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 22:57:19.197530 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.197508    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:19.274860 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.274831    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-openshift-ca\") pod \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") "
Oct 29 22:57:19.275058 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.274920    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-xks-ca\") pod \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") "
Oct 29 22:57:19.275181 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.275148    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-kube-api-access-f5l7m\" (UniqueName: \"kubernetes.io/projected/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-kube-api-access-f5l7m\") pod \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\" (UID: \"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1\") "
Oct 29 22:57:19.276015 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.275988    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-openshift-ca" pod "2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1" (UID: "2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:19.278975 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.278940    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-kube-api-access-f5l7m" pod "2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1" (UID: "2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1"). InnerVolumeSpecName "kube-api-access-f5l7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:19.279298 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.279247    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-xks-ca" pod "2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1" (UID: "2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:19.377270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.377234    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:19.377270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.377264    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:19.377270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.377274    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5l7m\" (UniqueName: \"kubernetes.io/projected/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1-kube-api-access-f5l7m\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:19.613295 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.613252    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"]
Oct 29 22:57:19.613829 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.613817    4742 cpu_manager.go:414] "containerMap: removing container" podUID="2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1" containerName="revoke-keys"
Oct 29 22:57:19.613873 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.613831    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1" containerName="revoke-keys"
Oct 29 22:57:19.613930 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.613921    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1" containerName="revoke-keys"
Oct 29 22:57:19.617058 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.617038    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.636147 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.636123    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"]
Oct 29 22:57:19.681415 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.681377    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.681599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.681580    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.681706 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.681691    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wq8h\" (UniqueName: \"kubernetes.io/projected/ff524b22-dd9b-4916-8024-e6acb9c39697-kube-api-access-2wq8h\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.786366 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.786331    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wq8h\" (UniqueName: \"kubernetes.io/projected/ff524b22-dd9b-4916-8024-e6acb9c39697-kube-api-access-2wq8h\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.787865 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.786751    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.788296 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.788272    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.789357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.789332    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.791713 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.791688    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.806535 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.806512    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wq8h\" (UniqueName: \"kubernetes.io/projected/ff524b22-dd9b-4916-8024-e6acb9c39697-kube-api-access-2wq8h\") pod \"maas-api-revoke-sub-c09b16bf-z4t9q\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:19.930761 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:19.930682    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:20.053937 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:20.053905    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"
Oct 29 22:57:20.056214 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:20.056179    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch" event={"ID":"2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1","Type":"ContainerDied","Data":"ae9171432136fdfdd5322d2464a0aeea3a744be9354014d6d225e03d0d76d0ea"}
Oct 29 22:57:20.056352 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:20.056242    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae9171432136fdfdd5322d2464a0aeea3a744be9354014d6d225e03d0d76d0ea"
Oct 29 22:57:20.240888 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:20.240858    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff524b22_dd9b_4916_8024_e6acb9c39697.slice/crio-131af25627f65d7e1494204b0ca0076a2eade92b0c41f15919b88d5322d74e50 WatchSource:0}: Error finding container 131af25627f65d7e1494204b0ca0076a2eade92b0c41f15919b88d5322d74e50: Status 404 returned error can't find the container with id 131af25627f65d7e1494204b0ca0076a2eade92b0c41f15919b88d5322d74e50
Oct 29 22:57:20.241471 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:20.241446    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"]
Oct 29 22:57:21.065480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:21.065448    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q" event={"ID":"ff524b22-dd9b-4916-8024-e6acb9c39697","Type":"ContainerStarted","Data":"27864d3d035a58f9e7bddb4592ea25d4d20460a9c1e05502b96885c2982563bb"}
Oct 29 22:57:21.065676 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:21.065497    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q" event={"ID":"ff524b22-dd9b-4916-8024-e6acb9c39697","Type":"ContainerStarted","Data":"131af25627f65d7e1494204b0ca0076a2eade92b0c41f15919b88d5322d74e50"}
Oct 29 22:57:21.137101 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:21.137045    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q" podStartSLOduration=2.137030601 podStartE2EDuration="2.137030601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:21.134885506 +0000 UTC m=+898.275627090" watchObservedRunningTime="2026-10-29 22:57:21.137030601 +0000 UTC m=+898.277772184"
Oct 29 22:57:22.071662 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:22.071625    4742 generic.go:356] "Generic (PLEG): container finished" podID="ff524b22-dd9b-4916-8024-e6acb9c39697" containerID="27864d3d035a58f9e7bddb4592ea25d4d20460a9c1e05502b96885c2982563bb" exitCode=0
Oct 29 22:57:22.072042 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:22.071704    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q" event={"ID":"ff524b22-dd9b-4916-8024-e6acb9c39697","Type":"ContainerDied","Data":"27864d3d035a58f9e7bddb4592ea25d4d20460a9c1e05502b96885c2982563bb"}
Oct 29 22:57:23.126148 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:57:23.126106    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:57:23.246063 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.246021    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:23.332904 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.332817    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff524b22-dd9b-4916-8024-e6acb9c39697-kube-api-access-2wq8h\" (UniqueName: \"kubernetes.io/projected/ff524b22-dd9b-4916-8024-e6acb9c39697-kube-api-access-2wq8h\") pod \"ff524b22-dd9b-4916-8024-e6acb9c39697\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") "
Oct 29 22:57:23.332904 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.332878    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-openshift-ca\") pod \"ff524b22-dd9b-4916-8024-e6acb9c39697\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") "
Oct 29 22:57:23.333161 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.332932    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-xks-ca\") pod \"ff524b22-dd9b-4916-8024-e6acb9c39697\" (UID: \"ff524b22-dd9b-4916-8024-e6acb9c39697\") "
Oct 29 22:57:23.333337 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.333313    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-openshift-ca" pod "ff524b22-dd9b-4916-8024-e6acb9c39697" (UID: "ff524b22-dd9b-4916-8024-e6acb9c39697"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:23.335197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.335158    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff524b22-dd9b-4916-8024-e6acb9c39697-kube-api-access-2wq8h" pod "ff524b22-dd9b-4916-8024-e6acb9c39697" (UID: "ff524b22-dd9b-4916-8024-e6acb9c39697"). InnerVolumeSpecName "kube-api-access-2wq8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:23.335197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.335181    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-xks-ca" pod "ff524b22-dd9b-4916-8024-e6acb9c39697" (UID: "ff524b22-dd9b-4916-8024-e6acb9c39697"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:23.434666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.434633    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wq8h\" (UniqueName: \"kubernetes.io/projected/ff524b22-dd9b-4916-8024-e6acb9c39697-kube-api-access-2wq8h\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:23.434666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.434663    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:23.434666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.434673    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff524b22-dd9b-4916-8024-e6acb9c39697-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:23.481199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.481167    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"]
Oct 29 22:57:23.481724 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.481710    4742 cpu_manager.go:414] "containerMap: removing container" podUID="ff524b22-dd9b-4916-8024-e6acb9c39697" containerName="revoke-keys"
Oct 29 22:57:23.481766 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.481726    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff524b22-dd9b-4916-8024-e6acb9c39697" containerName="revoke-keys"
Oct 29 22:57:23.481847 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.481838    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff524b22-dd9b-4916-8024-e6acb9c39697" containerName="revoke-keys"
Oct 29 22:57:23.485312 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.485295    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.504270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.504237    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"]
Oct 29 22:57:23.552209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.551960    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkfpg\" (UniqueName: \"kubernetes.io/projected/a06313a9-f24b-4c7d-aaa3-79351596a292-kube-api-access-qkfpg\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.552406 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.552307    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.552495 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.552474    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.653901 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.653861    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.654123 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.653990    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkfpg\" (UniqueName: \"kubernetes.io/projected/a06313a9-f24b-4c7d-aaa3-79351596a292-kube-api-access-qkfpg\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.654123 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.654074    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.654727 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.654702    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.656294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.656273    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.675099 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.675063    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkfpg\" (UniqueName: \"kubernetes.io/projected/a06313a9-f24b-4c7d-aaa3-79351596a292-kube-api-access-qkfpg\") pod \"maas-api-revoke-sub-ba7ef576-p94bm\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:23.799382 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:23.799349    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:24.087984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:24.087958    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"
Oct 29 22:57:24.088176 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:24.087977    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q" event={"ID":"ff524b22-dd9b-4916-8024-e6acb9c39697","Type":"ContainerDied","Data":"131af25627f65d7e1494204b0ca0076a2eade92b0c41f15919b88d5322d74e50"}
Oct 29 22:57:24.088176 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:24.088026    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="131af25627f65d7e1494204b0ca0076a2eade92b0c41f15919b88d5322d74e50"
Oct 29 22:57:24.129735 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:24.129703    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06313a9_f24b_4c7d_aaa3_79351596a292.slice/crio-6a85ada026d2beacaaf23604d856e11b0d884c38aa5c882906b5cc7bb7ec131c WatchSource:0}: Error finding container 6a85ada026d2beacaaf23604d856e11b0d884c38aa5c882906b5cc7bb7ec131c: Status 404 returned error can't find the container with id 6a85ada026d2beacaaf23604d856e11b0d884c38aa5c882906b5cc7bb7ec131c
Oct 29 22:57:24.130066 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:24.129784    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"]
Oct 29 22:57:25.095794 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.095764    4742 generic.go:356] "Generic (PLEG): container finished" podID="a06313a9-f24b-4c7d-aaa3-79351596a292" containerID="8f50ca82b11f16dd2552442f7b267eec556be06e7366964ecb7b260880596740" exitCode=0
Oct 29 22:57:25.095945 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.095818    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm" event={"ID":"a06313a9-f24b-4c7d-aaa3-79351596a292","Type":"ContainerDied","Data":"8f50ca82b11f16dd2552442f7b267eec556be06e7366964ecb7b260880596740"}
Oct 29 22:57:25.095945 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.095855    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm" event={"ID":"a06313a9-f24b-4c7d-aaa3-79351596a292","Type":"ContainerStarted","Data":"6a85ada026d2beacaaf23604d856e11b0d884c38aa5c882906b5cc7bb7ec131c"}
Oct 29 22:57:25.787886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.787847    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"]
Oct 29 22:57:25.792390 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.792367    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.835133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.835067    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"]
Oct 29 22:57:25.884513 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884483    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.884711 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884539    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.884711 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884560    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.884711 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884692    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.884858 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884745    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060de561-99f2-4378-b674-47d1e223a1c8-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.884858 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884829    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.884925 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884909    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.884956 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884933    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060de561-99f2-4378-b674-47d1e223a1c8-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.884989 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.884966    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdkq\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-kube-api-access-vbdkq\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986371 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986336    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986371 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986372    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986414    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986447    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060de561-99f2-4378-b674-47d1e223a1c8-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986476    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986549    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986580    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060de561-99f2-4378-b674-47d1e223a1c8-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986884 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986633    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdkq\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-kube-api-access-vbdkq\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.986884 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.986806    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.987119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.987052    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.987221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.987172    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.987285 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.987246    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.987285 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.987267    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.987503 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.987480    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060de561-99f2-4378-b674-47d1e223a1c8-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.989360 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.989340    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:25.989539 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:25.989523    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060de561-99f2-4378-b674-47d1e223a1c8-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:26.005403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.005373    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:26.005583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.005564    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdkq\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-kube-api-access-vbdkq\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:26.107648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.107621    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:26.261320 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.261285    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:26.358046 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.357966    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"]
Oct 29 22:57:26.364517 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.361735    4742 cpu_manager.go:414] "containerMap: removing container" podUID="a06313a9-f24b-4c7d-aaa3-79351596a292" containerName="revoke-keys"
Oct 29 22:57:26.364517 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.361757    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06313a9-f24b-4c7d-aaa3-79351596a292" containerName="revoke-keys"
Oct 29 22:57:26.364517 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.362522    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06313a9-f24b-4c7d-aaa3-79351596a292" containerName="revoke-keys"
Oct 29 22:57:26.368112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.368074    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.393199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.393170    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-xks-ca\") pod \"a06313a9-f24b-4c7d-aaa3-79351596a292\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") "
Oct 29 22:57:26.393401 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.393375    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06313a9-f24b-4c7d-aaa3-79351596a292-kube-api-access-qkfpg\" (UniqueName: \"kubernetes.io/projected/a06313a9-f24b-4c7d-aaa3-79351596a292-kube-api-access-qkfpg\") pod \"a06313a9-f24b-4c7d-aaa3-79351596a292\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") "
Oct 29 22:57:26.393497 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.393481    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-openshift-ca\") pod \"a06313a9-f24b-4c7d-aaa3-79351596a292\" (UID: \"a06313a9-f24b-4c7d-aaa3-79351596a292\") "
Oct 29 22:57:26.394759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.394732    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-openshift-ca" pod "a06313a9-f24b-4c7d-aaa3-79351596a292" (UID: "a06313a9-f24b-4c7d-aaa3-79351596a292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:26.395415 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.395304    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:26.399183 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.399158    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06313a9-f24b-4c7d-aaa3-79351596a292-kube-api-access-qkfpg" pod "a06313a9-f24b-4c7d-aaa3-79351596a292" (UID: "a06313a9-f24b-4c7d-aaa3-79351596a292"). InnerVolumeSpecName "kube-api-access-qkfpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:26.399612 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.399581    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"]
Oct 29 22:57:26.400589 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.400569    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-xks-ca" pod "a06313a9-f24b-4c7d-aaa3-79351596a292" (UID: "a06313a9-f24b-4c7d-aaa3-79351596a292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:26.458997 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:26.458967    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060de561_99f2_4378_b674_47d1e223a1c8.slice/crio-1b17ac7ca6cd805154414d9ab87c69309f739040d60ad86c5c27d0ef74d1e354 WatchSource:0}: Error finding container 1b17ac7ca6cd805154414d9ab87c69309f739040d60ad86c5c27d0ef74d1e354: Status 404 returned error can't find the container with id 1b17ac7ca6cd805154414d9ab87c69309f739040d60ad86c5c27d0ef74d1e354
Oct 29 22:57:26.459131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.459080    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"]
Oct 29 22:57:26.462162 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.462072    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:26.462256 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.462201    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:26.462256 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.462234    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:26.496292 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.496261    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.496446 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.496317    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.496588 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.496511    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4124490-af35-452c-8b90-2d878d194412-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.496588 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.496551    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4124490-af35-452c-8b90-2d878d194412-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.496771 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.496740    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.496883 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.496795    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.496883 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.496823    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.496883 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.496852    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.497033 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.497012    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrnp\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-kube-api-access-csrnp\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.497170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.497158    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkfpg\" (UniqueName: \"kubernetes.io/projected/a06313a9-f24b-4c7d-aaa3-79351596a292-kube-api-access-qkfpg\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:26.497205 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.497176    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06313a9-f24b-4c7d-aaa3-79351596a292-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:26.598757 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.598722    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrnp\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-kube-api-access-csrnp\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.598939 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.598909    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599015 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.598997    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599205 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.599184    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4124490-af35-452c-8b90-2d878d194412-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.599223    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4124490-af35-452c-8b90-2d878d194412-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599358 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.599324    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599358 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.599348    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.599360    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.599374    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599630 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.599604    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.599710 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.599691    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.600114 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.600069    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4124490-af35-452c-8b90-2d878d194412-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.600372 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.600328    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.600475 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.600372    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.601967 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.601943    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.602130 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.602071    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4124490-af35-452c-8b90-2d878d194412-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.618652 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.618573    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.618909 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.618889    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrnp\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-kube-api-access-csrnp\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.694385 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.694343    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:26.777186 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.771273    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"]
Oct 29 22:57:26.777784 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.777760    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:26.793135 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.793105    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"]
Oct 29 22:57:26.903149 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.903064    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:26.903308 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.903194    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdlf\" (UniqueName: \"kubernetes.io/projected/a05c7896-82c6-4b1a-ac33-339d90924c67-kube-api-access-jsdlf\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:26.903357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:26.903327    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:27.005179 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.005141    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:27.005364 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.005187    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdlf\" (UniqueName: \"kubernetes.io/projected/a05c7896-82c6-4b1a-ac33-339d90924c67-kube-api-access-jsdlf\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:27.005364 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.005277    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:27.007488 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.006587    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:27.008658 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.008634    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:27.011609 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.011589    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"]
Oct 29 22:57:27.012485 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:27.012462    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4124490_af35_452c_8b90_2d878d194412.slice/crio-2776c9ae24a64294a140a9e41de265408a3fba62a75060db0f639dc00ded3588 WatchSource:0}: Error finding container 2776c9ae24a64294a140a9e41de265408a3fba62a75060db0f639dc00ded3588: Status 404 returned error can't find the container with id 2776c9ae24a64294a140a9e41de265408a3fba62a75060db0f639dc00ded3588
Oct 29 22:57:27.015640 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.015607    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:27.015749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.015687    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:27.015749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.015734    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:27.025560 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.025532    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdlf\" (UniqueName: \"kubernetes.io/projected/a05c7896-82c6-4b1a-ac33-339d90924c67-kube-api-access-jsdlf\") pod \"maas-api-revoke-sub-2024ac24-p8npk\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:27.094362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.094332    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:27.111393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.111361    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" event={"ID":"060de561-99f2-4378-b674-47d1e223a1c8","Type":"ContainerStarted","Data":"f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45"}
Oct 29 22:57:27.111500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.111429    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" event={"ID":"060de561-99f2-4378-b674-47d1e223a1c8","Type":"ContainerStarted","Data":"1b17ac7ca6cd805154414d9ab87c69309f739040d60ad86c5c27d0ef74d1e354"}
Oct 29 22:57:27.112781 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.112758    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" event={"ID":"b4124490-af35-452c-8b90-2d878d194412","Type":"ContainerStarted","Data":"2776c9ae24a64294a140a9e41de265408a3fba62a75060db0f639dc00ded3588"}
Oct 29 22:57:27.114129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.114074    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"
Oct 29 22:57:27.114239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.114109    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm" event={"ID":"a06313a9-f24b-4c7d-aaa3-79351596a292","Type":"ContainerDied","Data":"6a85ada026d2beacaaf23604d856e11b0d884c38aa5c882906b5cc7bb7ec131c"}
Oct 29 22:57:27.114239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.114230    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a85ada026d2beacaaf23604d856e11b0d884c38aa5c882906b5cc7bb7ec131c"
Oct 29 22:57:27.195880 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.195508    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podStartSLOduration=2.195490087 podStartE2EDuration="2.195490087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:27.170748896 +0000 UTC m=+904.311490482" watchObservedRunningTime="2026-10-29 22:57:27.195490087 +0000 UTC m=+904.336231670"
Oct 29 22:57:27.433895 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:27.433864    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"]
Oct 29 22:57:27.435282 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:27.435258    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05c7896_82c6_4b1a_ac33_339d90924c67.slice/crio-3ff4959bb6250bcef9dab5fd0c28468851d2def98e6845fd4d51d0f445eea4e2 WatchSource:0}: Error finding container 3ff4959bb6250bcef9dab5fd0c28468851d2def98e6845fd4d51d0f445eea4e2: Status 404 returned error can't find the container with id 3ff4959bb6250bcef9dab5fd0c28468851d2def98e6845fd4d51d0f445eea4e2
Oct 29 22:57:28.107978 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.107920    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:28.109488 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.109457    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:28.109664 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.109644    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:28.123829 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.123802    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk" event={"ID":"a05c7896-82c6-4b1a-ac33-339d90924c67","Type":"ContainerStarted","Data":"b74b4f57f17ba645652746e461d99afebe1bc25452ffc1515bc2c8406fdc5c32"}
Oct 29 22:57:28.123945 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.123858    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk" event={"ID":"a05c7896-82c6-4b1a-ac33-339d90924c67","Type":"ContainerStarted","Data":"3ff4959bb6250bcef9dab5fd0c28468851d2def98e6845fd4d51d0f445eea4e2"}
Oct 29 22:57:28.126200 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.126179    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" event={"ID":"b4124490-af35-452c-8b90-2d878d194412","Type":"ContainerStarted","Data":"77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4"}
Oct 29 22:57:28.197119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.194201    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk" podStartSLOduration=2.194181297 podStartE2EDuration="2.194181297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:28.175640334 +0000 UTC m=+905.316381921" watchObservedRunningTime="2026-10-29 22:57:28.194181297 +0000 UTC m=+905.334922873"
Oct 29 22:57:28.218330 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.218281    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podStartSLOduration=2.218265933 podStartE2EDuration="2.218265933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:28.217634717 +0000 UTC m=+905.358376332" watchObservedRunningTime="2026-10-29 22:57:28.218265933 +0000 UTC m=+905.359007517"
Oct 29 22:57:28.697547 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.697468    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:28.702235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.701875    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:28.702235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:28.701937    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:29.112109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:29.111791    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:29.112109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:29.111866    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:29.197110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:29.196481    4742 generic.go:356] "Generic (PLEG): container finished" podID="a05c7896-82c6-4b1a-ac33-339d90924c67" containerID="b74b4f57f17ba645652746e461d99afebe1bc25452ffc1515bc2c8406fdc5c32" exitCode=0
Oct 29 22:57:29.201119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:29.197892    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk" event={"ID":"a05c7896-82c6-4b1a-ac33-339d90924c67","Type":"ContainerDied","Data":"b74b4f57f17ba645652746e461d99afebe1bc25452ffc1515bc2c8406fdc5c32"}
Oct 29 22:57:29.697108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:29.696503    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:29.697108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:29.696567    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:30.119562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.119518    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:30.120159 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.120124    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:30.434935 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.434574    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.663107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.662209    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.663318 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.663171    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.701632 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.701532    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:30.702301 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.701868    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.702942 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.701954    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:30.887247 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.887194    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.922111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.905204    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.922111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.917264    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.922111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.918109    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:30.922111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.918413    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:57:30.944112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.941157    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:30.944112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.941515    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:57:30.944112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.941760    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:57:30.944112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:30.942018    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:57:31.144172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.119488    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:31.144172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.119553    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:31.144172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.122147    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:57:31.674210 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.674166    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:31.729809 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.729779    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:57:31.732186 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.732151    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:31.846116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.842794    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:57:31.858144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.858113    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:31.858435 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.858368    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:31.893610 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.890412    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:57:31.894752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.894724    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:57:31.931819 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.908763    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:57:31.931819 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.909187    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:57:32.016939 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:31.989327    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:32.018489 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.017388    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.023107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.018685    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.044189 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.023376    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:57:32.044189 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.023445    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:57:32.044189 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.024106    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.059836 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.058279    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:57:32.073252 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.060991    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.097653 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.077227    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.107110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.098778    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.107110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.099069    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.123113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.113420    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:57:32.123113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.113551    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:57:32.123113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.113591    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:57:32.123113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.113627    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:57:32.123113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.116580    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05c7896-82c6-4b1a-ac33-339d90924c67-kube-api-access-jsdlf\" (UniqueName: \"kubernetes.io/projected/a05c7896-82c6-4b1a-ac33-339d90924c67-kube-api-access-jsdlf\") pod \"a05c7896-82c6-4b1a-ac33-339d90924c67\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") "
Oct 29 22:57:32.123113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.116692    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-openshift-ca\") pod \"a05c7896-82c6-4b1a-ac33-339d90924c67\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") "
Oct 29 22:57:32.123113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.116788    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-xks-ca\") pod \"a05c7896-82c6-4b1a-ac33-339d90924c67\" (UID: \"a05c7896-82c6-4b1a-ac33-339d90924c67\") "
Oct 29 22:57:32.170177 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.148326    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-openshift-ca" pod "a05c7896-82c6-4b1a-ac33-339d90924c67" (UID: "a05c7896-82c6-4b1a-ac33-339d90924c67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:32.197109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.158786    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:32.265208 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.265162    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:32.277200 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.158119    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.358148 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.223210    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:32.358478 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.358453    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:57:32.428399 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.170776    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.761479 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.601512    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05c7896-82c6-4b1a-ac33-339d90924c67-kube-api-access-jsdlf" pod "a05c7896-82c6-4b1a-ac33-339d90924c67" (UID: "a05c7896-82c6-4b1a-ac33-339d90924c67"). InnerVolumeSpecName "kube-api-access-jsdlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:32.761479 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.687041    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsdlf\" (UniqueName: \"kubernetes.io/projected/a05c7896-82c6-4b1a-ac33-339d90924c67-kube-api-access-jsdlf\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:32.773114 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.769206    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.791117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.773373    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:32.791117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.773429    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:32.858108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.857062    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.886307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.886232    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-xks-ca" pod "a05c7896-82c6-4b1a-ac33-339d90924c67" (UID: "a05c7896-82c6-4b1a-ac33-339d90924c67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:32.921117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.893538    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.921117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.895023    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05c7896-82c6-4b1a-ac33-339d90924c67-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:32.921117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:32.919142    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.096667 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.050170    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.096667 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.053158    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.096667 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.053500    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.096667 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.077626    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.096667 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.078438    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.096667 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.079128    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.109144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.108445    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:33.109144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.108510    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:33.231151 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.202117    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.231151 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.202487    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.368906 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.368784    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"
Oct 29 22:57:33.410833 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.383894    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.697968 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.696719    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:33.697968 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:33.696846    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:34.007201 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.007070    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.033119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.018990    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.036245 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.033985    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.036683 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.034015    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.067110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.062562    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.067761 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.067723    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.081108 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:57:34.073433    4742 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.073s"
Oct 29 22:57:34.081495 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.081462    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk" event={"ID":"a05c7896-82c6-4b1a-ac33-339d90924c67","Type":"ContainerDied","Data":"3ff4959bb6250bcef9dab5fd0c28468851d2def98e6845fd4d51d0f445eea4e2"}
Oct 29 22:57:34.081641 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.081626    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ff4959bb6250bcef9dab5fd0c28468851d2def98e6845fd4d51d0f445eea4e2"
Oct 29 22:57:34.126111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.101140    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.126771 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.111450    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.152110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.119343    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.204222 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.204176    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.225109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.206029    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Oct 29 22:57:34.227648 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.218487    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.234347 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.218981    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.234894 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.226573    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:34.246265 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.246221    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:34.473785 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.473744    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.525364 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.525117    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:57:34.545972 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.545752    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.599029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.596155    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:57:34.599498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.599479    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:57:34.703961 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.703926    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:34.798965 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.798871    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:34.852221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.849344    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"]
Oct 29 22:57:34.852221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.850210    4742 cpu_manager.go:414] "containerMap: removing container" podUID="a05c7896-82c6-4b1a-ac33-339d90924c67" containerName="revoke-keys"
Oct 29 22:57:34.852221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.850229    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05c7896-82c6-4b1a-ac33-339d90924c67" containerName="revoke-keys"
Oct 29 22:57:34.852221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.850402    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05c7896-82c6-4b1a-ac33-339d90924c67" containerName="revoke-keys"
Oct 29 22:57:34.882646 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.882615    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:34.992126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:34.982346    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:35.143280 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.143241    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.143760 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.143732    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.144271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.144249    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.157129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.150330    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.157129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.151008    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:35.157129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.151040    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:35.157129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.152026    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.157129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.152925    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.157129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.153473    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.157129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.155517    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.211224 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.211178    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.211467 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.211446    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.220556 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.211777    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngbr\" (UniqueName: \"kubernetes.io/projected/a6ded313-855e-471d-b90a-80bb77379d21-kube-api-access-6ngbr\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.265116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.257391    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:35.311116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.308567    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"]
Oct 29 22:57:35.311116 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.308681    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:57:35.312031 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.311518    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:57:35.340155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.326115    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:57:35.340155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.327347    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngbr\" (UniqueName: \"kubernetes.io/projected/a6ded313-855e-471d-b90a-80bb77379d21-kube-api-access-6ngbr\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.340155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.327485    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.340155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.327525    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.340155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.334671    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.396928 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.396823    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.402076 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.402049    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:57:35.426510 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.423413    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:57:35.443120 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.442698    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:57:35.454508 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.453098    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngbr\" (UniqueName: \"kubernetes.io/projected/a6ded313-855e-471d-b90a-80bb77379d21-kube-api-access-6ngbr\") pod \"maas-api-revoke-sub-ac9c9aa3-gck8h\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.526850 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.526818    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:35.625489 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.625417    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:57:35.629017 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.628982    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:57:35.632251 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.632222    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:57:35.722698 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.720908    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:35.722698 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:35.720966    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:36.026026 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.022134    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"]
Oct 29 22:57:36.026483 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:36.026454    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ded313_855e_471d_b90a_80bb77379d21.slice/crio-069da840f8dc8df466b9dda7b5cab297329ae8f4ed5807564dff3d400bb630c0 WatchSource:0}: Error finding container 069da840f8dc8df466b9dda7b5cab297329ae8f4ed5807564dff3d400bb630c0: Status 404 returned error can't find the container with id 069da840f8dc8df466b9dda7b5cab297329ae8f4ed5807564dff3d400bb630c0
Oct 29 22:57:36.109217 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.109175    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.109395 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.109235    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:36.644185 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.644141    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h" event={"ID":"a6ded313-855e-471d-b90a-80bb77379d21","Type":"ContainerStarted","Data":"951ba2d476139e794c120631d754eec661ee96fd249576660c89b527893ef94a"}
Oct 29 22:57:36.644591 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.644207    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h" event={"ID":"a6ded313-855e-471d-b90a-80bb77379d21","Type":"ContainerStarted","Data":"069da840f8dc8df466b9dda7b5cab297329ae8f4ed5807564dff3d400bb630c0"}
Oct 29 22:57:36.695864 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.695714    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.695864 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.695778    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:36.699935 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.699882    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h" podStartSLOduration=2.699868822 podStartE2EDuration="2.699868822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:36.694899539 +0000 UTC m=+913.835641124" watchObservedRunningTime="2026-10-29 22:57:36.699868822 +0000 UTC m=+913.840610406"
Oct 29 22:57:36.880383 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.880347    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"]
Oct 29 22:57:36.884748 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.884723    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:36.903799 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.903733    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"]
Oct 29 22:57:36.965045 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.965005    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:36.965238 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.965106    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwxbs\" (UniqueName: \"kubernetes.io/projected/b7f4d451-021f-4db2-838d-1aae8ebeddca-kube-api-access-zwxbs\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:36.965302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:36.965255    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:37.067157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.067112    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:37.067320 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.067198    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwxbs\" (UniqueName: \"kubernetes.io/projected/b7f4d451-021f-4db2-838d-1aae8ebeddca-kube-api-access-zwxbs\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:37.067320 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.067272    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:37.068198 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.068170    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:37.071170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.071075    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:37.091668 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.091634    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwxbs\" (UniqueName: \"kubernetes.io/projected/b7f4d451-021f-4db2-838d-1aae8ebeddca-kube-api-access-zwxbs\") pod \"maas-api-revoke-sub-33426e78-8wxw2\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:37.108799 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.108764    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.108961 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.108839    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 22:57:37.200217 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.200121    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:37.552752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.551235    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"]
Oct 29 22:57:37.655313 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.655266    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2" event={"ID":"b7f4d451-021f-4db2-838d-1aae8ebeddca","Type":"ContainerStarted","Data":"6f9697118b921db153ab54d9acc0046b05063b1c42a3434796032680b57b0e85"}
Oct 29 22:57:37.657982 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.657951    4742 generic.go:356] "Generic (PLEG): container finished" podID="a6ded313-855e-471d-b90a-80bb77379d21" containerID="951ba2d476139e794c120631d754eec661ee96fd249576660c89b527893ef94a" exitCode=0
Oct 29 22:57:37.658130 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.657993    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h" event={"ID":"a6ded313-855e-471d-b90a-80bb77379d21","Type":"ContainerDied","Data":"951ba2d476139e794c120631d754eec661ee96fd249576660c89b527893ef94a"}
Oct 29 22:57:37.696143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.695285    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.696143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:37.695363    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 22:57:38.671665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:38.671625    4742 generic.go:356] "Generic (PLEG): container finished" podID="b7f4d451-021f-4db2-838d-1aae8ebeddca" containerID="1d33611f30fdd5e9dc9007b4e66faeab643fa3684f18ffb1ead1a3540991a147" exitCode=0
Oct 29 22:57:38.672177 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:38.671737    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2" event={"ID":"b7f4d451-021f-4db2-838d-1aae8ebeddca","Type":"ContainerDied","Data":"1d33611f30fdd5e9dc9007b4e66faeab643fa3684f18ffb1ead1a3540991a147"}
Oct 29 22:57:38.895136 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:38.895096    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:39.037433 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.037402    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ded313-855e-471d-b90a-80bb77379d21-kube-api-access-6ngbr\" (UniqueName: \"kubernetes.io/projected/a6ded313-855e-471d-b90a-80bb77379d21-kube-api-access-6ngbr\") pod \"a6ded313-855e-471d-b90a-80bb77379d21\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") "
Oct 29 22:57:39.037663 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.037640    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-xks-ca\") pod \"a6ded313-855e-471d-b90a-80bb77379d21\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") "
Oct 29 22:57:39.037920 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.037898    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-openshift-ca\") pod \"a6ded313-855e-471d-b90a-80bb77379d21\" (UID: \"a6ded313-855e-471d-b90a-80bb77379d21\") "
Oct 29 22:57:39.038574 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.038403    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-openshift-ca" pod "a6ded313-855e-471d-b90a-80bb77379d21" (UID: "a6ded313-855e-471d-b90a-80bb77379d21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:39.038574 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.038563    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:39.040364 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.040331    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ded313-855e-471d-b90a-80bb77379d21-kube-api-access-6ngbr" pod "a6ded313-855e-471d-b90a-80bb77379d21" (UID: "a6ded313-855e-471d-b90a-80bb77379d21"). InnerVolumeSpecName "kube-api-access-6ngbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:39.040687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.040661    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-xks-ca" pod "a6ded313-855e-471d-b90a-80bb77379d21" (UID: "a6ded313-855e-471d-b90a-80bb77379d21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:39.140497 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.140411    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngbr\" (UniqueName: \"kubernetes.io/projected/a6ded313-855e-471d-b90a-80bb77379d21-kube-api-access-6ngbr\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:39.140497 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.140440    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ded313-855e-471d-b90a-80bb77379d21-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:39.149242 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.149215    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:39.149370 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.149261    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:39.227742 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.227713    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:39.228219 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.228189    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:39.229162 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.229144    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 22:57:39.682818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.682784    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"
Oct 29 22:57:39.682818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.682810    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h" event={"ID":"a6ded313-855e-471d-b90a-80bb77379d21","Type":"ContainerDied","Data":"069da840f8dc8df466b9dda7b5cab297329ae8f4ed5807564dff3d400bb630c0"}
Oct 29 22:57:39.683398 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.682876    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="069da840f8dc8df466b9dda7b5cab297329ae8f4ed5807564dff3d400bb630c0"
Oct 29 22:57:39.736609 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.736487    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:39.736609 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.736552    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:39.863558 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.863519    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:39.878435 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.878404    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:39.878801 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.878771    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:39.879711 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:39.879688    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 22:57:40.054422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.054328    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f4d451-021f-4db2-838d-1aae8ebeddca-kube-api-access-zwxbs\" (UniqueName: \"kubernetes.io/projected/b7f4d451-021f-4db2-838d-1aae8ebeddca-kube-api-access-zwxbs\") pod \"b7f4d451-021f-4db2-838d-1aae8ebeddca\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") "
Oct 29 22:57:40.054694 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.054438    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-openshift-ca\") pod \"b7f4d451-021f-4db2-838d-1aae8ebeddca\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") "
Oct 29 22:57:40.054694 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.054613    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-xks-ca\") pod \"b7f4d451-021f-4db2-838d-1aae8ebeddca\" (UID: \"b7f4d451-021f-4db2-838d-1aae8ebeddca\") "
Oct 29 22:57:40.054872 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.054841    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-openshift-ca" pod "b7f4d451-021f-4db2-838d-1aae8ebeddca" (UID: "b7f4d451-021f-4db2-838d-1aae8ebeddca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:40.055367 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.055344    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:40.056742 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.056715    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f4d451-021f-4db2-838d-1aae8ebeddca-kube-api-access-zwxbs" pod "b7f4d451-021f-4db2-838d-1aae8ebeddca" (UID: "b7f4d451-021f-4db2-838d-1aae8ebeddca"). InnerVolumeSpecName "kube-api-access-zwxbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:40.057358 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.057335    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-xks-ca" pod "b7f4d451-021f-4db2-838d-1aae8ebeddca" (UID: "b7f4d451-021f-4db2-838d-1aae8ebeddca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:40.156976 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.156942    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwxbs\" (UniqueName: \"kubernetes.io/projected/b7f4d451-021f-4db2-838d-1aae8ebeddca-kube-api-access-zwxbs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:40.156976 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.156976    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f4d451-021f-4db2-838d-1aae8ebeddca-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:40.690927 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.690897    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"
Oct 29 22:57:40.691362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.690936    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2" event={"ID":"b7f4d451-021f-4db2-838d-1aae8ebeddca","Type":"ContainerDied","Data":"6f9697118b921db153ab54d9acc0046b05063b1c42a3434796032680b57b0e85"}
Oct 29 22:57:40.691362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.691017    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f9697118b921db153ab54d9acc0046b05063b1c42a3434796032680b57b0e85"
Oct 29 22:57:40.890624 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.890592    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"]
Oct 29 22:57:40.891171 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.891158    4742 cpu_manager.go:414] "containerMap: removing container" podUID="a6ded313-855e-471d-b90a-80bb77379d21" containerName="revoke-keys"
Oct 29 22:57:40.891221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.891172    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ded313-855e-471d-b90a-80bb77379d21" containerName="revoke-keys"
Oct 29 22:57:40.891221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.891207    4742 cpu_manager.go:414] "containerMap: removing container" podUID="b7f4d451-021f-4db2-838d-1aae8ebeddca" containerName="revoke-keys"
Oct 29 22:57:40.891221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.891213    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f4d451-021f-4db2-838d-1aae8ebeddca" containerName="revoke-keys"
Oct 29 22:57:40.891321 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.891312    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f4d451-021f-4db2-838d-1aae8ebeddca" containerName="revoke-keys"
Oct 29 22:57:40.891360 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.891326    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ded313-855e-471d-b90a-80bb77379d21" containerName="revoke-keys"
Oct 29 22:57:40.895169 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.895153    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.921357 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.921325    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"]
Oct 29 22:57:40.967218 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967114    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.967218 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967168    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.967450 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967242    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.967450 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967280    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.967450 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967409    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.967606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967471    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.967606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967497    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.967719 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967623    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6h4\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-kube-api-access-gq6h4\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:40.967719 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:40.967655    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.068528 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068492    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.068752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068540    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.068752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068568    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.068752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068598    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.068752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068659    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.068752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068714    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.068752 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068736    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.069167 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068832    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6h4\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-kube-api-access-gq6h4\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.069167 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.068866    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.069167 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.069066    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.069167 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.069137    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.069392 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.069197    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.069392 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.069314    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.069392 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.069377    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.071027 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.071006    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.071203 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.071185    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.088886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.088859    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6h4\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-kube-api-access-gq6h4\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.088886 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.088877    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.212187 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.212152    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:41.534219 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.534191    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"]
Oct 29 22:57:41.534710 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:41.534683    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f0e924_1d9f_43a0_8a63_aaebf5ecc3d2.slice/crio-32d830dfce82f2337cbdeeab97b0db9d21bf940238e94da695663d0e94e00b91 WatchSource:0}: Error finding container 32d830dfce82f2337cbdeeab97b0db9d21bf940238e94da695663d0e94e00b91: Status 404 returned error can't find the container with id 32d830dfce82f2337cbdeeab97b0db9d21bf940238e94da695663d0e94e00b91
Oct 29 22:57:41.537072 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.537038    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:41.537172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.537122    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:41.537172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.537158    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:41.700502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.700467    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" event={"ID":"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2","Type":"ContainerStarted","Data":"946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561"}
Oct 29 22:57:41.700954 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.700523    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" event={"ID":"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2","Type":"ContainerStarted","Data":"32d830dfce82f2337cbdeeab97b0db9d21bf940238e94da695663d0e94e00b91"}
Oct 29 22:57:41.755413 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:41.755347    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podStartSLOduration=1.755332005 podStartE2EDuration="1.755332005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:41.752589069 +0000 UTC m=+918.893330665" watchObservedRunningTime="2026-10-29 22:57:41.755332005 +0000 UTC m=+918.896073591"
Oct 29 22:57:42.212346 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:42.212309    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:42.213678 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:42.213651    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:42.213819 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:42.213701    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 29 22:57:43.213583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:43.213547    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:43.213977 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:43.213611    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 29 22:57:44.212945 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:44.212912    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:44.213149 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:44.212966    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 29 22:57:45.214058 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:45.214022    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:45.214483 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:45.214081    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 29 22:57:46.213592 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:46.213555    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:46.213793 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:46.213618    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 29 22:57:47.212479 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:47.212439    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:47.212903 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:47.212507    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 29 22:57:48.212993 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.212958    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:48.213528 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.213024    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 29 22:57:48.477773 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.477700    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"]
Oct 29 22:57:48.482157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.482140    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.509945 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.509908    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:48.510178 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.510025    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:48.516868 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.516836    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"]
Oct 29 22:57:48.593836 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.593801    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrnqk\" (UniqueName: \"kubernetes.io/projected/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-kube-api-access-rrnqk\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.593836 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.593839    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.594135 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.594102    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.695155 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.695078    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.695344 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.695280    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrnqk\" (UniqueName: \"kubernetes.io/projected/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-kube-api-access-rrnqk\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.695435 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.695404    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.696131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.696103    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.697632 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.697608    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.715361 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.715334    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrnqk\" (UniqueName: \"kubernetes.io/projected/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-kube-api-access-rrnqk\") pod \"maas-api-revoke-sub-25637484-nhkhv\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:48.797595 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:48.797500    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:49.122450 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:49.122419    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"]
Oct 29 22:57:49.123628 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:49.123593    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e5badd_a4a2_43fe_bbab_a1d5249ad4a5.slice/crio-2bc0bf050bddf0221422df196b41f1761702900d4cbe08d4021bb3abe671b047 WatchSource:0}: Error finding container 2bc0bf050bddf0221422df196b41f1761702900d4cbe08d4021bb3abe671b047: Status 404 returned error can't find the container with id 2bc0bf050bddf0221422df196b41f1761702900d4cbe08d4021bb3abe671b047
Oct 29 22:57:49.217272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:49.217237    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:49.217673 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:49.217634    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:49.218466 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:49.218450    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 22:57:49.755845 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:49.755757    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv" event={"ID":"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5","Type":"ContainerStarted","Data":"a875c0ecd6390143f7b31788a625678d519299ad855e2edc82adef6f06d0239a"}
Oct 29 22:57:49.755845 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:49.755824    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv" event={"ID":"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5","Type":"ContainerStarted","Data":"2bc0bf050bddf0221422df196b41f1761702900d4cbe08d4021bb3abe671b047"}
Oct 29 22:57:49.795247 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:49.795194    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv" podStartSLOduration=1.795179131 podStartE2EDuration="1.795179131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:49.79397846 +0000 UTC m=+926.934720036" watchObservedRunningTime="2026-10-29 22:57:49.795179131 +0000 UTC m=+926.935920715"
Oct 29 22:57:50.764714 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:50.764681    4742 generic.go:356] "Generic (PLEG): container finished" podID="92e5badd-a4a2-43fe-bbab-a1d5249ad4a5" containerID="a875c0ecd6390143f7b31788a625678d519299ad855e2edc82adef6f06d0239a" exitCode=0
Oct 29 22:57:50.765225 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:50.764756    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv" event={"ID":"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5","Type":"ContainerDied","Data":"a875c0ecd6390143f7b31788a625678d519299ad855e2edc82adef6f06d0239a"}
Oct 29 22:57:51.928755 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:51.928730    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:52.044165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.043988    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-openshift-ca\") pod \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") "
Oct 29 22:57:52.044165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.044058    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-kube-api-access-rrnqk\" (UniqueName: \"kubernetes.io/projected/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-kube-api-access-rrnqk\") pod \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") "
Oct 29 22:57:52.044165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.044140    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-xks-ca\") pod \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\" (UID: \"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5\") "
Oct 29 22:57:52.044661 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.044428    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-openshift-ca" pod "92e5badd-a4a2-43fe-bbab-a1d5249ad4a5" (UID: "92e5badd-a4a2-43fe-bbab-a1d5249ad4a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:52.044940 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.044914    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:52.046273 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.046250    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-kube-api-access-rrnqk" pod "92e5badd-a4a2-43fe-bbab-a1d5249ad4a5" (UID: "92e5badd-a4a2-43fe-bbab-a1d5249ad4a5"). InnerVolumeSpecName "kube-api-access-rrnqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:52.046498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.046483    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-xks-ca" pod "92e5badd-a4a2-43fe-bbab-a1d5249ad4a5" (UID: "92e5badd-a4a2-43fe-bbab-a1d5249ad4a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:52.146192 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.146148    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrnqk\" (UniqueName: \"kubernetes.io/projected/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-kube-api-access-rrnqk\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:52.146192 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.146188    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:57:52.779162 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.779114    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv" event={"ID":"92e5badd-a4a2-43fe-bbab-a1d5249ad4a5","Type":"ContainerDied","Data":"2bc0bf050bddf0221422df196b41f1761702900d4cbe08d4021bb3abe671b047"}
Oct 29 22:57:52.779375 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.779201    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bc0bf050bddf0221422df196b41f1761702900d4cbe08d4021bb3abe671b047"
Oct 29 22:57:52.779375 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:52.779132    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"
Oct 29 22:57:53.124043 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:57:53.124009    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:57:56.381339 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.381309    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"]
Oct 29 22:57:56.381918 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.381902    4742 cpu_manager.go:414] "containerMap: removing container" podUID="92e5badd-a4a2-43fe-bbab-a1d5249ad4a5" containerName="revoke-keys"
Oct 29 22:57:56.381962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.381921    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e5badd-a4a2-43fe-bbab-a1d5249ad4a5" containerName="revoke-keys"
Oct 29 22:57:56.382020 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.382011    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e5badd-a4a2-43fe-bbab-a1d5249ad4a5" containerName="revoke-keys"
Oct 29 22:57:56.385747 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.385719    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.427165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.427130    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"]
Oct 29 22:57:56.503291 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503258    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.503492 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503297    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.503492 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503457    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.503621 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503502    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.503621 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503613    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85fs\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-kube-api-access-b85fs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.503740 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503644    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.503916 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503896    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.503978 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503946    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.504036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.503982    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.605715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.605679    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.605715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.605722    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.605961 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.605759    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85fs\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-kube-api-access-b85fs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.605961 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.605788    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.605961    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606006    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606028    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606078    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606122    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606402    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606528 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606476    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606656 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606538    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606725 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606698    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.606771 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.606723    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.608162 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.608138    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.608285 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.608271    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.627784 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.627759    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.628038 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.628015    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85fs\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-kube-api-access-b85fs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:56.710199 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:56.710118    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:57.058609 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:57.058557    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"]
Oct 29 22:57:57.086609 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:57:57.086572    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aea0d04_5bfa_41eb_a0ad_5b5187e5cf3a.slice/crio-d80df30bcdc7919d532a43dd3620864046480af13eccd1f3714bb1a17eafd0fc WatchSource:0}: Error finding container d80df30bcdc7919d532a43dd3620864046480af13eccd1f3714bb1a17eafd0fc: Status 404 returned error can't find the container with id d80df30bcdc7919d532a43dd3620864046480af13eccd1f3714bb1a17eafd0fc
Oct 29 22:57:57.090938 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:57.089990    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:57.090938 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:57.090101    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:57.090938 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:57.090151    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:57:57.888119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:57.883602    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" event={"ID":"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a","Type":"ContainerStarted","Data":"0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd"}
Oct 29 22:57:57.888119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:57.883683    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" event={"ID":"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a","Type":"ContainerStarted","Data":"d80df30bcdc7919d532a43dd3620864046480af13eccd1f3714bb1a17eafd0fc"}
Oct 29 22:57:57.967111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:57.965554    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podStartSLOduration=1.965530313 podStartE2EDuration="1.965530313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:57.946568084 +0000 UTC m=+935.087309668" watchObservedRunningTime="2026-10-29 22:57:57.965530313 +0000 UTC m=+935.106271898"
Oct 29 22:57:58.713351 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:58.713318    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:57:58.713703 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:58.713680    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:57:58.713868 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:58.713843    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:57:59.149141 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.133609    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.659779 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.153192    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.659779 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.206648    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.659779 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.227194    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.659779 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.418181    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.659779 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.535483    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.659779 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.535579    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.659779 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.538772    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.915746 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.915616    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:57:59.916025 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.915976    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:00.019107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:57:59.994195    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded"
Oct 29 22:58:00.040385 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.009158    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.126909 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.098261    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.256168 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.231079    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.328160 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.302160    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.344188 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:58:00.310984    4742 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.311s"
Oct 29 22:58:00.435110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.406253    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"]
Oct 29 22:58:00.556170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.555287    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.673302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.637193    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.673302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.638347    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.673302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.664137    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.673302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.665730    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.673302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.665770    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.673302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.671132    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.673661 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.665730    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.696111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.689227    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.696111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.692270    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.783575 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.783534    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:58:00.783864 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.783833    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:00.788445 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.788407    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:58:00.788599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.788478    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:58:00.788599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.788519    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:58:00.788599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.788553    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:58:00.788599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.788587    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:58:00.788833 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.788625    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:58:00.788833 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.788660    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:58:00.788833 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.788695    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:58:00.801723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.794071    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:00.801723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.797759    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"]
Oct 29 22:58:00.801723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.800577    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:00.801723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.800615    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:00.801723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.800733    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh79w\" (UniqueName: \"kubernetes.io/projected/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-kube-api-access-kh79w\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:00.944279 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.944239    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:01.017172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:00.953790    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:01.088171 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.084997    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:01.088171 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.085048    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:01.088171 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.085162    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh79w\" (UniqueName: \"kubernetes.io/projected/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-kube-api-access-kh79w\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:01.089704 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.089659    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:01.163150 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.163105    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.164599 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.164552    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.211889 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.211844    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:01.214461 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.214361    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh79w\" (UniqueName: \"kubernetes.io/projected/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-kube-api-access-kh79w\") pod \"maas-api-revoke-sub-2a8622e1-j6sfp\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:01.299365 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.299332    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:58:01.300723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.300014    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:58:01.301986 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.301969    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:58:01.330855 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.330813    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.341198 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.341100    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:58:01.342952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.342921    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.344678 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.344294    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:01.370657 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.344856    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:58:01.685312 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.685264    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.784187 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.770452    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.784187 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.687660    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.784187 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.705292    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.784187 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.712551    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:58:01.785874 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.688587    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.788335 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.713073    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.788489 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.767406    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:58:01.788489 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:01.788416    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:02.136137 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.125594    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 29 22:58:02.181030 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.180988    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.181615 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.181586    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.218388 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.212538    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.354028 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.353979    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.368110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.357967    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.766168 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.758237    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.793748 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.793707    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:02.802212 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.785783    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.824176 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.785751    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.836115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.785809    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.844376 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.785829    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.869119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.793314    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.877124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.793363    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.971618 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:02.785862    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.215613 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.215569    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.262112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.245471    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": context deadline exceeded"
Oct 29 22:58:03.262112 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.252868    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.445798 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.445277    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.445798 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.445473    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.449170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.446385    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:58:03.449170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.446439    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:58:03.449170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.446472    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:58:03.449170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.446505    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:58:03.449170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.446537    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:58:03.449170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.446566    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:58:03.449170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.447221    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:58:03.453063 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.452509    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"]
Oct 29 22:58:03.458215 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.458188    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:58:03.504487 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.504464    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:58:03.524148 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.524121    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:58:03.544849 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.544815    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:58:03.697443 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.697268    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp" event={"ID":"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d","Type":"ContainerStarted","Data":"3a8774d656fd5a79053aad16d84130510f3a77adc2dd6abc4cf68a2d0ae1d8cf"}
Oct 29 22:58:03.719594 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.711372    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:58:03.719594 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:03.711436    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:04.707015 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:04.706984    4742 generic.go:356] "Generic (PLEG): container finished" podID="1b0fc532-f498-4b81-9be4-8d0ba34d3f9d" containerID="7b68d3a6b90db0b2e8da6ff441c909a688f532616cd8da832df0432d68405d0a" exitCode=0
Oct 29 22:58:04.707424 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:04.707059    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp" event={"ID":"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d","Type":"ContainerDied","Data":"7b68d3a6b90db0b2e8da6ff441c909a688f532616cd8da832df0432d68405d0a"}
Oct 29 22:58:04.711317 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:04.711288    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:58:04.711458 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:04.711341    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:05.711120 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:05.710986    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:58:05.711579 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:05.711072    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:06.004602 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.004527    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:06.136856 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.118759    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-kube-api-access-kh79w\" (UniqueName: \"kubernetes.io/projected/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-kube-api-access-kh79w\") pod \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") "
Oct 29 22:58:06.136856 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.118820    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-openshift-ca\") pod \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") "
Oct 29 22:58:06.136856 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.118958    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-xks-ca\") pod \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\" (UID: \"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d\") "
Oct 29 22:58:06.137689 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.137623    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-openshift-ca" pod "1b0fc532-f498-4b81-9be4-8d0ba34d3f9d" (UID: "1b0fc532-f498-4b81-9be4-8d0ba34d3f9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:06.141351 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.141299    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-xks-ca" pod "1b0fc532-f498-4b81-9be4-8d0ba34d3f9d" (UID: "1b0fc532-f498-4b81-9be4-8d0ba34d3f9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:06.142645 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.142611    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-kube-api-access-kh79w" pod "1b0fc532-f498-4b81-9be4-8d0ba34d3f9d" (UID: "1b0fc532-f498-4b81-9be4-8d0ba34d3f9d"). InnerVolumeSpecName "kube-api-access-kh79w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:06.221491 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.221454    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh79w\" (UniqueName: \"kubernetes.io/projected/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-kube-api-access-kh79w\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:06.221491 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.221491    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:06.221706 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.221507    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:06.711221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.711185    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:58:06.711604 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.711240    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:06.722916 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.722883    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp" event={"ID":"1b0fc532-f498-4b81-9be4-8d0ba34d3f9d","Type":"ContainerDied","Data":"3a8774d656fd5a79053aad16d84130510f3a77adc2dd6abc4cf68a2d0ae1d8cf"}
Oct 29 22:58:06.722916 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.722916    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"
Oct 29 22:58:06.723125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.722958    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8774d656fd5a79053aad16d84130510f3a77adc2dd6abc4cf68a2d0ae1d8cf"
Oct 29 22:58:06.923578 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.923544    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"]
Oct 29 22:58:06.924144 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.924129    4742 cpu_manager.go:414] "containerMap: removing container" podUID="1b0fc532-f498-4b81-9be4-8d0ba34d3f9d" containerName="revoke-keys"
Oct 29 22:58:06.924203 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.924145    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0fc532-f498-4b81-9be4-8d0ba34d3f9d" containerName="revoke-keys"
Oct 29 22:58:06.924270 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.924260    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0fc532-f498-4b81-9be4-8d0ba34d3f9d" containerName="revoke-keys"
Oct 29 22:58:06.927754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.927738    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:06.943451 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.943426    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:06.943617 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.943603    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:06.946874 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:06.946848    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"]
Oct 29 22:58:07.032823 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.032720    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.033080 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.033058    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.033287 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.033257    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r7vt\" (UniqueName: \"kubernetes.io/projected/1d3ec66b-f795-49b6-8b9a-fe0a43168784-kube-api-access-2r7vt\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.134474 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.134432    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.134474 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.134480    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.134689 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.134517    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7vt\" (UniqueName: \"kubernetes.io/projected/1d3ec66b-f795-49b6-8b9a-fe0a43168784-kube-api-access-2r7vt\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.135170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.135151    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.136875 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.136847    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.154366 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.154322    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7vt\" (UniqueName: \"kubernetes.io/projected/1d3ec66b-f795-49b6-8b9a-fe0a43168784-kube-api-access-2r7vt\") pod \"maas-api-revoke-sub-bac3c53e-dxmxm\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.241444 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.241400    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:07.567909 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:58:07.567872    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3ec66b_f795_49b6_8b9a_fe0a43168784.slice/crio-01d8414e1f03087f2851331cda684fdeafca2222001dfe48289c52eef84a17fa WatchSource:0}: Error finding container 01d8414e1f03087f2851331cda684fdeafca2222001dfe48289c52eef84a17fa: Status 404 returned error can't find the container with id 01d8414e1f03087f2851331cda684fdeafca2222001dfe48289c52eef84a17fa
Oct 29 22:58:07.571186 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.571155    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"]
Oct 29 22:58:07.710749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.710710    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:58:07.710950 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.710777    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 22:58:07.756154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:07.756049    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm" event={"ID":"1d3ec66b-f795-49b6-8b9a-fe0a43168784","Type":"ContainerStarted","Data":"01d8414e1f03087f2851331cda684fdeafca2222001dfe48289c52eef84a17fa"}
Oct 29 22:58:08.748399 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:08.748299    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:58:08.749422 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:08.749137    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:58:08.750650 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:08.750627    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 22:58:08.764656 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:08.764627    4742 generic.go:356] "Generic (PLEG): container finished" podID="1d3ec66b-f795-49b6-8b9a-fe0a43168784" containerID="66673b571b75a680404a931a13a93c45a4f5ecd8857692fb2eea13916349bf04" exitCode=0
Oct 29 22:58:08.765055 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:08.764711    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm" event={"ID":"1d3ec66b-f795-49b6-8b9a-fe0a43168784","Type":"ContainerDied","Data":"66673b571b75a680404a931a13a93c45a4f5ecd8857692fb2eea13916349bf04"}
Oct 29 22:58:09.952170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:09.952144    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:10.079223 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.079140    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-openshift-ca\") pod \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") "
Oct 29 22:58:10.079223 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.079218    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-xks-ca\") pod \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") "
Oct 29 22:58:10.079518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.079500    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3ec66b-f795-49b6-8b9a-fe0a43168784-kube-api-access-2r7vt\" (UniqueName: \"kubernetes.io/projected/1d3ec66b-f795-49b6-8b9a-fe0a43168784-kube-api-access-2r7vt\") pod \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\" (UID: \"1d3ec66b-f795-49b6-8b9a-fe0a43168784\") "
Oct 29 22:58:10.079600 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.079578    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-openshift-ca" pod "1d3ec66b-f795-49b6-8b9a-fe0a43168784" (UID: "1d3ec66b-f795-49b6-8b9a-fe0a43168784"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:10.080114 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.080053    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:10.081330 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.081297    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-xks-ca" pod "1d3ec66b-f795-49b6-8b9a-fe0a43168784" (UID: "1d3ec66b-f795-49b6-8b9a-fe0a43168784"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:10.081581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.081561    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3ec66b-f795-49b6-8b9a-fe0a43168784-kube-api-access-2r7vt" pod "1d3ec66b-f795-49b6-8b9a-fe0a43168784" (UID: "1d3ec66b-f795-49b6-8b9a-fe0a43168784"). InnerVolumeSpecName "kube-api-access-2r7vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:10.181938 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.181900    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r7vt\" (UniqueName: \"kubernetes.io/projected/1d3ec66b-f795-49b6-8b9a-fe0a43168784-kube-api-access-2r7vt\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:10.181938 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.181933    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3ec66b-f795-49b6-8b9a-fe0a43168784-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:10.782853 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.782824    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"
Oct 29 22:58:10.783047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.782852    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm" event={"ID":"1d3ec66b-f795-49b6-8b9a-fe0a43168784","Type":"ContainerDied","Data":"01d8414e1f03087f2851331cda684fdeafca2222001dfe48289c52eef84a17fa"}
Oct 29 22:58:10.783047 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:10.782910    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01d8414e1f03087f2851331cda684fdeafca2222001dfe48289c52eef84a17fa"
Oct 29 22:58:16.915701 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:16.915664    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"]
Oct 29 22:58:16.916434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:16.916413    4742 cpu_manager.go:414] "containerMap: removing container" podUID="1d3ec66b-f795-49b6-8b9a-fe0a43168784" containerName="revoke-keys"
Oct 29 22:58:16.916493 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:16.916439    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3ec66b-f795-49b6-8b9a-fe0a43168784" containerName="revoke-keys"
Oct 29 22:58:16.916666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:16.916655    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3ec66b-f795-49b6-8b9a-fe0a43168784" containerName="revoke-keys"
Oct 29 22:58:16.920719 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:16.920698    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:16.952868 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:16.952830    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"]
Oct 29 22:58:16.968943 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:16.968911    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2bd3f07e-openshift-default-dockercfg-6rh2f\""
Oct 29 22:58:17.084494 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.084439    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-token\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.084891 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.084852    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdcmb\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-kube-api-access-xdcmb\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.085009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.084903    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-certs\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.085009 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.084933    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-envoy\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.085182 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.085113    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-data\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.085248 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.085193    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-podinfo\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.085248 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.085235    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e35b87-8978-47a2-b4d5-423f76d30f21-istiod-ca-cert\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.085350 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.085266    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-credential-socket\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.085408 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.085355    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-socket\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187054 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.186953    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-data\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187296 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.187067    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-podinfo\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187296 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.187153    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e35b87-8978-47a2-b4d5-423f76d30f21-istiod-ca-cert\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187296 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.187189    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-credential-socket\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187296 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.187226    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-socket\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.187345    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-token\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.187395    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcmb\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-kube-api-access-xdcmb\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.187437    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-certs\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.187516 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.187467    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-envoy\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.188715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.188645    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-credential-socket\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.188883 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.188865    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-data\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.192028 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.191964    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-certs\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.194839 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.194807    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-socket\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.195583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.195557    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e35b87-8978-47a2-b4d5-423f76d30f21-istiod-ca-cert\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.195808 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.195782    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-envoy\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.197304 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.196821    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-podinfo\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.219559 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.219454    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcmb\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-kube-api-access-xdcmb\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.220263 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.220237    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-token\") pod \"e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") " pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.234334 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.234306    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:17.593132 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.593101    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"]
Oct 29 22:58:17.593296 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:58:17.593274    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e35b87_8978_47a2_b4d5_423f76d30f21.slice/crio-eae189186b242bba0031985930a10769a520ca5023275a6dd65b2bf4cc4b4441 WatchSource:0}: Error finding container eae189186b242bba0031985930a10769a520ca5023275a6dd65b2bf4cc4b4441: Status 404 returned error can't find the container with id eae189186b242bba0031985930a10769a520ca5023275a6dd65b2bf4cc4b4441
Oct 29 22:58:17.595701 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.595671    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:58:17.595754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.595735    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:58:17.595794 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.595765    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:58:17.780219 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.780186    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"]
Oct 29 22:58:17.784432 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.784412    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:17.806658 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.806628    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:17.806853 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.806729    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:17.810628 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.810591    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"]
Oct 29 22:58:17.841157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.840260    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n" event={"ID":"06e35b87-8978-47a2-b4d5-423f76d30f21","Type":"ContainerStarted","Data":"9b18a87c0c22a5e294c798cf813287ba3bf4e140278d35c558ad02576cf7579e"}
Oct 29 22:58:17.841157 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.840330    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n" event={"ID":"06e35b87-8978-47a2-b4d5-423f76d30f21","Type":"ContainerStarted","Data":"eae189186b242bba0031985930a10769a520ca5023275a6dd65b2bf4cc4b4441"}
Oct 29 22:58:17.893909 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.893853    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n" podStartSLOduration=1.8938395099999998 podStartE2EDuration="1.89383951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:17.890395855 +0000 UTC m=+955.031137439" watchObservedRunningTime="2026-10-29 22:58:17.89383951 +0000 UTC m=+955.034581094"
Oct 29 22:58:17.898230 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.898206    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxkl\" (UniqueName: \"kubernetes.io/projected/671014f5-a911-41e0-927f-200bdcb80a43-kube-api-access-jrxkl\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:17.898355 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.898332    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:17.898534 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.898520    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:17.999341 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.999305    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:17.999810 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.999594    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:17.999810 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:17.999682    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxkl\" (UniqueName: \"kubernetes.io/projected/671014f5-a911-41e0-927f-200bdcb80a43-kube-api-access-jrxkl\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:18.000115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.000073    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:18.001915 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.001895    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:18.024302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.024266    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxkl\" (UniqueName: \"kubernetes.io/projected/671014f5-a911-41e0-927f-200bdcb80a43-kube-api-access-jrxkl\") pod \"maas-api-revoke-sub-987324d0-7jcd6\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:18.101237 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.101138    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:18.234798 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.234765    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:18.240234 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.240202    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:18.414427 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.414388    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"]
Oct 29 22:58:18.414590 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:58:18.414565    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671014f5_a911_41e0_927f_200bdcb80a43.slice/crio-a0319826274288763eae893dd7d0b9c0ebb5feb78d53b9453850a7c8fa8da32d WatchSource:0}: Error finding container a0319826274288763eae893dd7d0b9c0ebb5feb78d53b9453850a7c8fa8da32d: Status 404 returned error can't find the container with id a0319826274288763eae893dd7d0b9c0ebb5feb78d53b9453850a7c8fa8da32d
Oct 29 22:58:18.849006 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.848968    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6" event={"ID":"671014f5-a911-41e0-927f-200bdcb80a43","Type":"ContainerStarted","Data":"1612ebfe60b884aebaa56f9e2ac838a9daae0d66e29da53a1f609ea3466599c0"}
Oct 29 22:58:18.849196 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.849029    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6" event={"ID":"671014f5-a911-41e0-927f-200bdcb80a43","Type":"ContainerStarted","Data":"a0319826274288763eae893dd7d0b9c0ebb5feb78d53b9453850a7c8fa8da32d"}
Oct 29 22:58:18.849245 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.849208    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:18.850373 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.850358    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:58:18.889687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:18.889627    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6" podStartSLOduration=1.889608928 podStartE2EDuration="1.889608928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:18.887947535 +0000 UTC m=+956.028689119" watchObservedRunningTime="2026-10-29 22:58:18.889608928 +0000 UTC m=+956.030350513"
Oct 29 22:58:19.869552 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:19.869517    4742 generic.go:356] "Generic (PLEG): container finished" podID="671014f5-a911-41e0-927f-200bdcb80a43" containerID="1612ebfe60b884aebaa56f9e2ac838a9daae0d66e29da53a1f609ea3466599c0" exitCode=0
Oct 29 22:58:19.870066 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:19.869604    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6" event={"ID":"671014f5-a911-41e0-927f-200bdcb80a43","Type":"ContainerDied","Data":"1612ebfe60b884aebaa56f9e2ac838a9daae0d66e29da53a1f609ea3466599c0"}
Oct 29 22:58:21.045224 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.045200    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:21.148182 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.148053    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671014f5-a911-41e0-927f-200bdcb80a43-kube-api-access-jrxkl\" (UniqueName: \"kubernetes.io/projected/671014f5-a911-41e0-927f-200bdcb80a43-kube-api-access-jrxkl\") pod \"671014f5-a911-41e0-927f-200bdcb80a43\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") "
Oct 29 22:58:21.148351 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.148193    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-openshift-ca\") pod \"671014f5-a911-41e0-927f-200bdcb80a43\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") "
Oct 29 22:58:21.148351 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.148216    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-xks-ca\") pod \"671014f5-a911-41e0-927f-200bdcb80a43\" (UID: \"671014f5-a911-41e0-927f-200bdcb80a43\") "
Oct 29 22:58:21.148589 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.148564    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-openshift-ca" pod "671014f5-a911-41e0-927f-200bdcb80a43" (UID: "671014f5-a911-41e0-927f-200bdcb80a43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:21.148769 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.148751    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:21.150165 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.150135    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671014f5-a911-41e0-927f-200bdcb80a43-kube-api-access-jrxkl" pod "671014f5-a911-41e0-927f-200bdcb80a43" (UID: "671014f5-a911-41e0-927f-200bdcb80a43"). InnerVolumeSpecName "kube-api-access-jrxkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:21.150262 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.150190    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-xks-ca" pod "671014f5-a911-41e0-927f-200bdcb80a43" (UID: "671014f5-a911-41e0-927f-200bdcb80a43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:21.250912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.250875    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrxkl\" (UniqueName: \"kubernetes.io/projected/671014f5-a911-41e0-927f-200bdcb80a43-kube-api-access-jrxkl\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:21.250912 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.250915    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/671014f5-a911-41e0-927f-200bdcb80a43-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:21.887628 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.887592    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6" event={"ID":"671014f5-a911-41e0-927f-200bdcb80a43","Type":"ContainerDied","Data":"a0319826274288763eae893dd7d0b9c0ebb5feb78d53b9453850a7c8fa8da32d"}
Oct 29 22:58:21.887952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.887651    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0319826274288763eae893dd7d0b9c0ebb5feb78d53b9453850a7c8fa8da32d"
Oct 29 22:58:21.887952 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:21.887604    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"
Oct 29 22:58:23.137002 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:58:23.136938    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:58:24.249202 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.248134    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"]
Oct 29 22:58:24.296111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.257005    4742 cpu_manager.go:414] "containerMap: removing container" podUID="671014f5-a911-41e0-927f-200bdcb80a43" containerName="revoke-keys"
Oct 29 22:58:24.347190 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.347142    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="671014f5-a911-41e0-927f-200bdcb80a43" containerName="revoke-keys"
Oct 29 22:58:24.347730 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.347695    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="671014f5-a911-41e0-927f-200bdcb80a43" containerName="revoke-keys"
Oct 29 22:58:24.439111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.413419    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.483109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.415184    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"]
Oct 29 22:58:24.483408 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.432056    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634172d1-c4be-49b8-993d-7fcbc9792698-istio-podinfo\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.483603 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.483583    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-istio-token\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.483810 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.483790    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-socket\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.483981 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.483962    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-envoy\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.484143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.484123    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-data\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.484299 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.484282    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-credential-socket\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.484459 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.484441    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-certs\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.484685 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.484667    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634172d1-c4be-49b8-993d-7fcbc9792698-istiod-ca-cert\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.484864 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.484845    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgl7\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-kube-api-access-wsgl7\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.485857 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.465700    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.487624 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.476016    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-01e33b3d-gw-openshift-default-dockercfg-646dz\""
Oct 29 22:58:24.533963 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.512253    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.561457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.560200    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.564103 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.561719    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.596746 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.596710    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-credential-socket\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.597036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597010    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-certs\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.597272 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597253    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634172d1-c4be-49b8-993d-7fcbc9792698-istiod-ca-cert\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597362    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-credential-socket\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597433    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgl7\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-kube-api-access-wsgl7\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597553    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634172d1-c4be-49b8-993d-7fcbc9792698-istio-podinfo\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597604    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-istio-token\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597686    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-socket\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597748    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-envoy\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.597785    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-data\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.598131    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634172d1-c4be-49b8-993d-7fcbc9792698-istiod-ca-cert\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.598156    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-data\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.598403    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-certs\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.609154 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.598718    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-socket\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.694115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.691034    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgl7\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-kube-api-access-wsgl7\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.705256 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.700881    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-istio-token\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.741502 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.741445    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-envoy\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.756131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.742408    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634172d1-c4be-49b8-993d-7fcbc9792698-istio-podinfo\") pod \"e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") " pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:24.900255 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.900209    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.917835 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.917782    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:24.918143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.918113    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.965247 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:24.965205    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:25.690724 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:25.690686    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"]
Oct 29 22:58:25.692023 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:25.691980    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:58:25.692294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:25.692269    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:58:25.692433 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:25.692416    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:58:26.243639 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:26.243600    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" event={"ID":"634172d1-c4be-49b8-993d-7fcbc9792698","Type":"ContainerStarted","Data":"6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d"}
Oct 29 22:58:26.243819 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:26.243664    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" event={"ID":"634172d1-c4be-49b8-993d-7fcbc9792698","Type":"ContainerStarted","Data":"f7e0d41f71d8fa5d302c6904e9b878b972741198cb5f2c81b9c8c6e10a121729"}
Oct 29 22:58:26.312302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:26.312239    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" podStartSLOduration=3.312219446 podStartE2EDuration="3.312219446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:26.309218195 +0000 UTC m=+963.449959785" watchObservedRunningTime="2026-10-29 22:58:26.312219446 +0000 UTC m=+963.452961032"
Oct 29 22:58:26.966203 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:26.966169    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:26.971473 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:26.971449    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:27.251391 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:27.251317    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:27.252206 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:27.252185    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:58:29.835606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:29.835573    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"]
Oct 29 22:58:29.840827 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:29.840806    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:29.858431 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:29.858402    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"]
Oct 29 22:58:29.863456 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:29.862885    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:29.863675 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:29.863654    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:29.901389 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:29.901352    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:29.901595 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:29.901557    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlhx5\" (UniqueName: \"kubernetes.io/projected/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-kube-api-access-mlhx5\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:29.901706 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:29.901690    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:30.003713 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:30.003679    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlhx5\" (UniqueName: \"kubernetes.io/projected/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-kube-api-access-mlhx5\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:30.003895 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:30.003797    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:30.003946 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:30.003920    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:30.004611 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:30.004583    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:30.006316 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:30.006292    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:30.052607 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:30.052572    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlhx5\" (UniqueName: \"kubernetes.io/projected/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-kube-api-access-mlhx5\") pod \"maas-api-revoke-sub-8d35c3c6-dtcmk\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:30.159142 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:30.159027    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:30.589759 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:30.589730    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"]
Oct 29 22:58:30.589869 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:58:30.589842    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d34d281_1cf8_473a_a5c5_47a7f8351ac3.slice/crio-4bf70b52a709ca4fa1550dd8dd5b4fb84c7f90292c29b66c6a3c80fe27e2efe9 WatchSource:0}: Error finding container 4bf70b52a709ca4fa1550dd8dd5b4fb84c7f90292c29b66c6a3c80fe27e2efe9: Status 404 returned error can't find the container with id 4bf70b52a709ca4fa1550dd8dd5b4fb84c7f90292c29b66c6a3c80fe27e2efe9
Oct 29 22:58:31.287095 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:31.286984    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk" event={"ID":"4d34d281-1cf8-473a-a5c5-47a7f8351ac3","Type":"ContainerStarted","Data":"527fc4f43d9d81399af6e54d6fc6da702d4f424f00daeb5ec274244275c456ba"}
Oct 29 22:58:31.287095 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:31.287038    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk" event={"ID":"4d34d281-1cf8-473a-a5c5-47a7f8351ac3","Type":"ContainerStarted","Data":"4bf70b52a709ca4fa1550dd8dd5b4fb84c7f90292c29b66c6a3c80fe27e2efe9"}
Oct 29 22:58:31.331393 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:31.331340    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk" podStartSLOduration=2.331324741 podStartE2EDuration="2.331324741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:31.327416591 +0000 UTC m=+968.468158175" watchObservedRunningTime="2026-10-29 22:58:31.331324741 +0000 UTC m=+968.472066325"
Oct 29 22:58:32.296785 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:32.296746    4742 generic.go:356] "Generic (PLEG): container finished" podID="4d34d281-1cf8-473a-a5c5-47a7f8351ac3" containerID="527fc4f43d9d81399af6e54d6fc6da702d4f424f00daeb5ec274244275c456ba" exitCode=0
Oct 29 22:58:32.297223 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:32.296837    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk" event={"ID":"4d34d281-1cf8-473a-a5c5-47a7f8351ac3","Type":"ContainerDied","Data":"527fc4f43d9d81399af6e54d6fc6da702d4f424f00daeb5ec274244275c456ba"}
Oct 29 22:58:33.477643 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.477614    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:33.587733 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.587701    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-openshift-ca\") pod \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") "
Oct 29 22:58:33.587925 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.587752    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-xks-ca\") pod \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") "
Oct 29 22:58:33.588010 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.587993    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-kube-api-access-mlhx5\" (UniqueName: \"kubernetes.io/projected/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-kube-api-access-mlhx5\") pod \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\" (UID: \"4d34d281-1cf8-473a-a5c5-47a7f8351ac3\") "
Oct 29 22:58:33.588180 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.588058    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-openshift-ca" pod "4d34d281-1cf8-473a-a5c5-47a7f8351ac3" (UID: "4d34d281-1cf8-473a-a5c5-47a7f8351ac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:33.588834 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.588807    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:33.590200 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.590169    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-kube-api-access-mlhx5" pod "4d34d281-1cf8-473a-a5c5-47a7f8351ac3" (UID: "4d34d281-1cf8-473a-a5c5-47a7f8351ac3"). InnerVolumeSpecName "kube-api-access-mlhx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:33.590381 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.590359    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-xks-ca" pod "4d34d281-1cf8-473a-a5c5-47a7f8351ac3" (UID: "4d34d281-1cf8-473a-a5c5-47a7f8351ac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:33.689975 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.689943    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:33.689975 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:33.689972    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlhx5\" (UniqueName: \"kubernetes.io/projected/4d34d281-1cf8-473a-a5c5-47a7f8351ac3-kube-api-access-mlhx5\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:34.311876 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:34.311846    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"
Oct 29 22:58:34.312056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:34.311884    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk" event={"ID":"4d34d281-1cf8-473a-a5c5-47a7f8351ac3","Type":"ContainerDied","Data":"4bf70b52a709ca4fa1550dd8dd5b4fb84c7f90292c29b66c6a3c80fe27e2efe9"}
Oct 29 22:58:34.312056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:34.311929    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bf70b52a709ca4fa1550dd8dd5b4fb84c7f90292c29b66c6a3c80fe27e2efe9"
Oct 29 22:58:37.577707 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.577667    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"]
Oct 29 22:58:37.578586 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.578563    4742 cpu_manager.go:414] "containerMap: removing container" podUID="4d34d281-1cf8-473a-a5c5-47a7f8351ac3" containerName="revoke-keys"
Oct 29 22:58:37.578586 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.578585    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d34d281-1cf8-473a-a5c5-47a7f8351ac3" containerName="revoke-keys"
Oct 29 22:58:37.578768 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.578683    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d34d281-1cf8-473a-a5c5-47a7f8351ac3" containerName="revoke-keys"
Oct 29 22:58:37.582831 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.582811    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.610495 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.610454    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:37.610668 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.610455    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:37.618294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.618265    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"]
Oct 29 22:58:37.649007 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.648967    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.649365 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.649333    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.649583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.649558    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p2dn\" (UniqueName: \"kubernetes.io/projected/013e3ace-4b65-460d-8208-d58e4a8210ec-kube-api-access-2p2dn\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.751454 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.751417    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2dn\" (UniqueName: \"kubernetes.io/projected/013e3ace-4b65-460d-8208-d58e4a8210ec-kube-api-access-2p2dn\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.751662 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.751552    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.751662 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.751648    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.752271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.752244    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.754232 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.754207    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.771433 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.771408    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2dn\" (UniqueName: \"kubernetes.io/projected/013e3ace-4b65-460d-8208-d58e4a8210ec-kube-api-access-2p2dn\") pod \"maas-api-revoke-sub-76c941b5-bbmqn\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:37.897301 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:37.897266    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:38.200375 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:58:38.200345    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013e3ace_4b65_460d_8208_d58e4a8210ec.slice/crio-6b0f2916d559ef986369eb6310b2085b973e9fb437a4c41baa91176c93743fc6 WatchSource:0}: Error finding container 6b0f2916d559ef986369eb6310b2085b973e9fb437a4c41baa91176c93743fc6: Status 404 returned error can't find the container with id 6b0f2916d559ef986369eb6310b2085b973e9fb437a4c41baa91176c93743fc6
Oct 29 22:58:38.200937 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:38.200911    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"]
Oct 29 22:58:38.344527 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:38.344479    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn" event={"ID":"013e3ace-4b65-460d-8208-d58e4a8210ec","Type":"ContainerStarted","Data":"6b0f2916d559ef986369eb6310b2085b973e9fb437a4c41baa91176c93743fc6"}
Oct 29 22:58:39.352990 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:39.352952    4742 generic.go:356] "Generic (PLEG): container finished" podID="013e3ace-4b65-460d-8208-d58e4a8210ec" containerID="e23640795f556d68be41f43e5dbefd5e1c01081ee422f42b7ef21cbd733416a7" exitCode=0
Oct 29 22:58:39.353390 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:39.353034    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn" event={"ID":"013e3ace-4b65-460d-8208-d58e4a8210ec","Type":"ContainerDied","Data":"e23640795f556d68be41f43e5dbefd5e1c01081ee422f42b7ef21cbd733416a7"}
Oct 29 22:58:40.764552 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:40.764524    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:40.915597 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:40.915500    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-openshift-ca\") pod \"013e3ace-4b65-460d-8208-d58e4a8210ec\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") "
Oct 29 22:58:40.916754 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:40.916729    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013e3ace-4b65-460d-8208-d58e4a8210ec-kube-api-access-2p2dn\" (UniqueName: \"kubernetes.io/projected/013e3ace-4b65-460d-8208-d58e4a8210ec-kube-api-access-2p2dn\") pod \"013e3ace-4b65-460d-8208-d58e4a8210ec\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") "
Oct 29 22:58:40.917499 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:40.917476    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-xks-ca\") pod \"013e3ace-4b65-460d-8208-d58e4a8210ec\" (UID: \"013e3ace-4b65-460d-8208-d58e4a8210ec\") "
Oct 29 22:58:40.917726 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:40.916635    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-openshift-ca" pod "013e3ace-4b65-460d-8208-d58e4a8210ec" (UID: "013e3ace-4b65-460d-8208-d58e4a8210ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:40.918603 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:40.918560    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:40.920628 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:40.920582    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013e3ace-4b65-460d-8208-d58e4a8210ec-kube-api-access-2p2dn" pod "013e3ace-4b65-460d-8208-d58e4a8210ec" (UID: "013e3ace-4b65-460d-8208-d58e4a8210ec"). InnerVolumeSpecName "kube-api-access-2p2dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:40.927571 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:40.927544    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-xks-ca" pod "013e3ace-4b65-460d-8208-d58e4a8210ec" (UID: "013e3ace-4b65-460d-8208-d58e4a8210ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:41.043837 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:41.043751    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p2dn\" (UniqueName: \"kubernetes.io/projected/013e3ace-4b65-460d-8208-d58e4a8210ec-kube-api-access-2p2dn\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:41.043837 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:41.043807    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013e3ace-4b65-460d-8208-d58e4a8210ec-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:41.429064 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:41.427962    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn" event={"ID":"013e3ace-4b65-460d-8208-d58e4a8210ec","Type":"ContainerDied","Data":"6b0f2916d559ef986369eb6310b2085b973e9fb437a4c41baa91176c93743fc6"}
Oct 29 22:58:41.429064 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:41.428026    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0f2916d559ef986369eb6310b2085b973e9fb437a4c41baa91176c93743fc6"
Oct 29 22:58:41.429064 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:41.428178    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"
Oct 29 22:58:42.181109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.181032    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:58:42.226406 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.225190    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:58:42.226406 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.226263    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.343109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.337282    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.381107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.358193    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.447557 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.447451    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:58:42.506503 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.506465    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:42.506788 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.506759    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.513325 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.513282    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.551148 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.549680    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.551148 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.550052    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:58:42.798190 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.794715    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.824108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.821451    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.956117 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:42.952187    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.059175 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.049180    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.200415 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.200216    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.248107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.247490    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.248403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.248371    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.316173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.298484    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:58:43.316173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.298550    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:58:43.316173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.298589    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:58:43.343125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.338326    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.343125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.339055    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:58:43.386674 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.385220    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.386674 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.385662    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:58:43.452110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.448164    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.452110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.448983    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:58:43.622110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.594294    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.622110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.594704    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.622110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.595003    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.622110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.595320    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:58:43.622110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.595377    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:58:43.622110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.595416    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:58:43.837172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.828487    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.837172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.829716    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:58:43.837172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.836236    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.837172 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:43.836519    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:58:44.108316 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.108277    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:44.138512 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.138475    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:58:44.150131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.145222    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:58:44.150131 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.145283    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:58:44.150731 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.150705    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:58:44.162050 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.162002    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:44.162812 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.162794    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:58:44.168352 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.168159    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:58:44.187405 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.187382    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:58:44.194307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.188816    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:58:44.194307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.190577    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:58:44.200035 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.200013    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:58:44.200293 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.200265    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:58:44.235836 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.235806    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:58:44.255136 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:44.254626    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:58:46.153230 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.153196    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"]
Oct 29 22:58:46.154022 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.154001    4742 cpu_manager.go:414] "containerMap: removing container" podUID="013e3ace-4b65-460d-8208-d58e4a8210ec" containerName="revoke-keys"
Oct 29 22:58:46.154022 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.154022    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013e3ace-4b65-460d-8208-d58e4a8210ec" containerName="revoke-keys"
Oct 29 22:58:46.154251 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.154191    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="013e3ace-4b65-460d-8208-d58e4a8210ec" containerName="revoke-keys"
Oct 29 22:58:46.158114 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.158094    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.184994 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.184965    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"]
Oct 29 22:58:46.190127 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.189601    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:46.190127 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.189745    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:46.200454 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.200426    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cf77\" (UniqueName: \"kubernetes.io/projected/d0af68dc-1260-4c55-a33d-352ba320d7b9-kube-api-access-5cf77\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.200591 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.200494    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.200591 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.200544    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.302562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.302520    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cf77\" (UniqueName: \"kubernetes.io/projected/d0af68dc-1260-4c55-a33d-352ba320d7b9-kube-api-access-5cf77\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.302769 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.302586    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.302769 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.302619    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.303312 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.303281    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.304997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.304978    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.324537 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.324507    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cf77\" (UniqueName: \"kubernetes.io/projected/d0af68dc-1260-4c55-a33d-352ba320d7b9-kube-api-access-5cf77\") pod \"maas-api-revoke-sub-3bf3cf70-rfl7q\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.474622 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.474541    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:46.786384 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:58:46.786349    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0af68dc_1260_4c55_a33d_352ba320d7b9.slice/crio-af9672037d424394fc6c623919bb6af48b70afa2d67a212b40ecd74507dc331e WatchSource:0}: Error finding container af9672037d424394fc6c623919bb6af48b70afa2d67a212b40ecd74507dc331e: Status 404 returned error can't find the container with id af9672037d424394fc6c623919bb6af48b70afa2d67a212b40ecd74507dc331e
Oct 29 22:58:46.787043 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:46.787021    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"]
Oct 29 22:58:47.051223 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.051136    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"]
Oct 29 22:58:47.061768 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.061741    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.093256 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.093221    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"]
Oct 29 22:58:47.096979 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.096952    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-01e33b3d-serving-cert\""
Oct 29 22:58:47.113855 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.113824    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-metrics-certs\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.114000 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.113894    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-maas-api-tls\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.114206 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.114176    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f165aa12-7e76-424b-b4ec-02b808b8b281-openshift-service-ca\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.114416 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.114394    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtrk\" (UniqueName: \"kubernetes.io/projected/f165aa12-7e76-424b-b4ec-02b808b8b281-kube-api-access-4jtrk\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.179103 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.179057    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q" event={"ID":"d0af68dc-1260-4c55-a33d-352ba320d7b9","Type":"ContainerStarted","Data":"87a8aaff55f83a73eb7ec5389206b54bd5ad4dc577fb8fc2fcf399542905f65e"}
Oct 29 22:58:47.179500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.179124    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q" event={"ID":"d0af68dc-1260-4c55-a33d-352ba320d7b9","Type":"ContainerStarted","Data":"af9672037d424394fc6c623919bb6af48b70afa2d67a212b40ecd74507dc331e"}
Oct 29 22:58:47.215917 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.215887    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-metrics-certs\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.216114 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.215950    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-maas-api-tls\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.216114 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.216036    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f165aa12-7e76-424b-b4ec-02b808b8b281-openshift-service-ca\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.216253 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.216233    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtrk\" (UniqueName: \"kubernetes.io/projected/f165aa12-7e76-424b-b4ec-02b808b8b281-kube-api-access-4jtrk\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.216733 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.216706    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f165aa12-7e76-424b-b4ec-02b808b8b281-openshift-service-ca\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.218468 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.218447    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-metrics-certs\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.218565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.218487    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-maas-api-tls\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.233842 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.233790    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q" podStartSLOduration=1.233776511 podStartE2EDuration="1.233776511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:47.230259354 +0000 UTC m=+984.371000939" watchObservedRunningTime="2026-10-29 22:58:47.233776511 +0000 UTC m=+984.374518094"
Oct 29 22:58:47.242996 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.242961    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtrk\" (UniqueName: \"kubernetes.io/projected/f165aa12-7e76-424b-b4ec-02b808b8b281-kube-api-access-4jtrk\") pod \"maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.377148 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.377116    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:47.725468 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.725438    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"]
Oct 29 22:58:47.761303 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:58:47.761067    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf165aa12_7e76_424b_b4ec_02b808b8b281.slice/crio-df8f4d59da00a52e6a3251ad0cad2523f46eb57b836c24d57a9547831aa97888 WatchSource:0}: Error finding container df8f4d59da00a52e6a3251ad0cad2523f46eb57b836c24d57a9547831aa97888: Status 404 returned error can't find the container with id df8f4d59da00a52e6a3251ad0cad2523f46eb57b836c24d57a9547831aa97888
Oct 29 22:58:47.763223 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:47.763195    4742 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:58:48.189593 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:48.189558    4742 generic.go:356] "Generic (PLEG): container finished" podID="d0af68dc-1260-4c55-a33d-352ba320d7b9" containerID="87a8aaff55f83a73eb7ec5389206b54bd5ad4dc577fb8fc2fcf399542905f65e" exitCode=0
Oct 29 22:58:48.189998 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:48.189640    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q" event={"ID":"d0af68dc-1260-4c55-a33d-352ba320d7b9","Type":"ContainerDied","Data":"87a8aaff55f83a73eb7ec5389206b54bd5ad4dc577fb8fc2fcf399542905f65e"}
Oct 29 22:58:48.191640 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:48.191617    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5" event={"ID":"f165aa12-7e76-424b-b4ec-02b808b8b281","Type":"ContainerStarted","Data":"df8f4d59da00a52e6a3251ad0cad2523f46eb57b836c24d57a9547831aa97888"}
Oct 29 22:58:49.216843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.216791    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5" event={"ID":"f165aa12-7e76-424b-b4ec-02b808b8b281","Type":"ContainerStarted","Data":"7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5"}
Oct 29 22:58:49.217313 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.217056    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:58:49.265978 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.265914    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5" podStartSLOduration=1.917194777 podStartE2EDuration="2.26589327s" totalImagesPullingTime="348.698493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:49.258819424 +0000 UTC m=+986.399561008" watchObservedRunningTime="2026-10-29 22:58:49.26589327 +0000 UTC m=+986.406634861"
Oct 29 22:58:49.373008 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.372980    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:49.453189 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.451186    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-xks-ca\") pod \"d0af68dc-1260-4c55-a33d-352ba320d7b9\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") "
Oct 29 22:58:49.453189 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.451244    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0af68dc-1260-4c55-a33d-352ba320d7b9-kube-api-access-5cf77\" (UniqueName: \"kubernetes.io/projected/d0af68dc-1260-4c55-a33d-352ba320d7b9-kube-api-access-5cf77\") pod \"d0af68dc-1260-4c55-a33d-352ba320d7b9\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") "
Oct 29 22:58:49.453189 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.451424    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-openshift-ca\") pod \"d0af68dc-1260-4c55-a33d-352ba320d7b9\" (UID: \"d0af68dc-1260-4c55-a33d-352ba320d7b9\") "
Oct 29 22:58:49.453189 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.452314    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-openshift-ca" pod "d0af68dc-1260-4c55-a33d-352ba320d7b9" (UID: "d0af68dc-1260-4c55-a33d-352ba320d7b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:49.456593 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.456570    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0af68dc-1260-4c55-a33d-352ba320d7b9-kube-api-access-5cf77" pod "d0af68dc-1260-4c55-a33d-352ba320d7b9" (UID: "d0af68dc-1260-4c55-a33d-352ba320d7b9"). InnerVolumeSpecName "kube-api-access-5cf77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:49.459951 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.459887    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-xks-ca" pod "d0af68dc-1260-4c55-a33d-352ba320d7b9" (UID: "d0af68dc-1260-4c55-a33d-352ba320d7b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:49.553997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.553959    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:49.553997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.553992    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cf77\" (UniqueName: \"kubernetes.io/projected/d0af68dc-1260-4c55-a33d-352ba320d7b9-kube-api-access-5cf77\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:49.553997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:49.554004    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0af68dc-1260-4c55-a33d-352ba320d7b9-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:58:50.225943 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:50.225915    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"
Oct 29 22:58:50.226362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:50.225953    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q" event={"ID":"d0af68dc-1260-4c55-a33d-352ba320d7b9","Type":"ContainerDied","Data":"af9672037d424394fc6c623919bb6af48b70afa2d67a212b40ecd74507dc331e"}
Oct 29 22:58:50.226362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:50.225996    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9672037d424394fc6c623919bb6af48b70afa2d67a212b40ecd74507dc331e"
Oct 29 22:58:53.124159 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:58:53.124120    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:58:54.851665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:54.851630    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"]
Oct 29 22:58:54.876739 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:54.876710    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"]
Oct 29 22:58:55.231452 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:58:55.231379    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:59:00.342018 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:00.341977    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"]
Oct 29 22:59:00.372500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:00.372467    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"]
Oct 29 22:59:04.120111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.119222    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Oct 29 22:59:04.162126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.146321    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:59:04.162126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.147792    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded"
Oct 29 22:59:04.162126 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.152387    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:59:04.170124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.168200    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.199819 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.196741    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:59:04.220142 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.200145    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.220142 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.218314    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.253111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.241330    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.253111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.241781    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.280594 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.276282    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:59:04.557170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.553546    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.619110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.603396    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:04.619110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.603501    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.619110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.603922    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:04.619110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.603996    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.759118 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.755652    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:04.759118 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:04.755721    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.194512 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.184927    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.228518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.209977    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:59:05.401111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.386190    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:05.401111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.386253    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.401111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.386554    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.453022 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.415202    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf" podUID="c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" containerName="payload-processing" containerID="cri-o://676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee" gracePeriod=25
Oct 29 22:59:05.453022 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.425360    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.453022 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.426051    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.453022 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.433821    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.600546 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.600337    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:05.728582 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.728444    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.822729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.822385    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-t7bjt" podUID="f56a8bcc-2991-4a16-891e-fa69e277ca6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.823193 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.823158    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:59:05.921110 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.896181    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:59:05.947151 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.910566    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:59:05.972111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.910607    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:59:05.972111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.959301    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:59:05.988624 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:05.988522    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798" podUID="5620b0a7-c9a8-4652-b202-d522ac93f01b" containerName="payload-pre-processing" containerID="cri-o://fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207" gracePeriod=25
Oct 29 22:59:06.199419 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.199375    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.217984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.217931    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.231797 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:06.231763    4742 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.084s"
Oct 29 22:59:06.232075 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.232060    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:59:06.232242 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.232230    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:59:06.232368 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.232358    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:59:06.232480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.232470    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:59:06.232610 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.232597    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:59:06.232772 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.232759    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:59:06.232911 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.232899    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:59:06.233029 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.233018    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:59:06.233168 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.233156    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:59:06.279227 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.279120    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.303151 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.302799    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw" podUID="b833cd5c-a426-4822-bcff-4965be0c17e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.429109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.403720    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.643111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.619540    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.643111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.619886    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.643111 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.634498    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.857119 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.855400    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.953306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.953200    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.954909 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.954876    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.955723 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.955704    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:59:06.957709 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.957685    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.959318 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:06.959298    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:59:07.022066 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.022024    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:59:07.221538 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.221430    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.312107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.287615    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.338390 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.296217    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.410972 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.410932    4742 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 29 22:59:07.636209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.605553    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.636209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.610380    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:59:07.636209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.610445    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 22:59:07.636209 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.629173    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.656731 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.652952    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:59:07.701296 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.658550    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.883317 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:07.856100    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:08.054163 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.029270    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:59:08.054163 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.030612    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7" podUID="2b671691-1df4-4fc2-9ba3-eb27464f3553" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:08.054163 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.030772    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 22:59:08.054163 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.031821    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:08.109108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.093663    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 22:59:08.109108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.103154    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 22:59:08.345162 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.340273    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:59:08.415324 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.415251    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:08.434793 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.434766    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7"
Oct 29 22:59:08.445115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.435830    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 22:59:08.445115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.437040    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:59:08.445115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.441880    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 22:59:08.445115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.442954    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 22:59:08.445115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.443022    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:59:08.445115 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.443075    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 22:59:08.457818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:08.457790    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 22:59:09.894217 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:09.894158    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main" containerID="cri-o://fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67" gracePeriod=45
Oct 29 22:59:09.898028 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:09.897958    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" containerID="cri-o://ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32" gracePeriod=45
Oct 29 22:59:10.231003 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.230966    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798" podUID="5620b0a7-c9a8-4652-b202-d522ac93f01b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:10.231469 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.230992    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf" podUID="c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:10.241766 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.241733    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:59:10.292108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.291712    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-home\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-home\") pod \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") "
Oct 29 22:59:10.292108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.291765    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tls-certs\") pod \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") "
Oct 29 22:59:10.292108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.291814    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tmp-dir\") pod \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") "
Oct 29 22:59:10.292108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.291865    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-dshm\") pod \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") "
Oct 29 22:59:10.292108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.292014    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-home" pod "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" (UID: "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.292108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.292059    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-model-cache\") pod \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") "
Oct 29 22:59:10.292562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.292220    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-kube-api-access-8n8z9\" (UniqueName: \"kubernetes.io/projected/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-kube-api-access-8n8z9\") pod \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\" (UID: \"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2\") "
Oct 29 22:59:10.292562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.292402    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tmp-dir" pod "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" (UID: "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.292562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.292418    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-model-cache" pod "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" (UID: "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.293027 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.292847    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.293027 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.292870    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.293027 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.292884    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.294766 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.294438    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tls-certs" pod "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" (UID: "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:10.295039 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.295016    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-kube-api-access-8n8z9" pod "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" (UID: "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2"). InnerVolumeSpecName "kube-api-access-8n8z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:10.295176 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.295153    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:59:10.295611 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.295587    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-dshm" pod "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" (UID: "5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.393814 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.393781    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tls-certs\") pod \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") "
Oct 29 22:59:10.394004 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.393831    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-dshm\") pod \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") "
Oct 29 22:59:10.394004 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.393880    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-home\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-home\") pod \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") "
Oct 29 22:59:10.394004 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.393964    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfce1f2-1214-4926-bfeb-5f651d4416d6-kube-api-access-4k57l\" (UniqueName: \"kubernetes.io/projected/1dfce1f2-1214-4926-bfeb-5f651d4416d6-kube-api-access-4k57l\") pod \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") "
Oct 29 22:59:10.394289 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394121    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-model-cache\") pod \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") "
Oct 29 22:59:10.394289 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394220    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-home" pod "1dfce1f2-1214-4926-bfeb-5f651d4416d6" (UID: "1dfce1f2-1214-4926-bfeb-5f651d4416d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.394289 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394245    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tmp-dir\") pod \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\" (UID: \"1dfce1f2-1214-4926-bfeb-5f651d4416d6\") "
Oct 29 22:59:10.394446 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394349    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-model-cache" pod "1dfce1f2-1214-4926-bfeb-5f651d4416d6" (UID: "1dfce1f2-1214-4926-bfeb-5f651d4416d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.394498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394462    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tmp-dir" pod "1dfce1f2-1214-4926-bfeb-5f651d4416d6" (UID: "1dfce1f2-1214-4926-bfeb-5f651d4416d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.394870 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394851    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n8z9\" (UniqueName: \"kubernetes.io/projected/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-kube-api-access-8n8z9\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.394962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394873    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.394962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394887    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.394962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394898    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.394962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394912    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.394962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.394925    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.396124 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.396106    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tls-certs" pod "1dfce1f2-1214-4926-bfeb-5f651d4416d6" (UID: "1dfce1f2-1214-4926-bfeb-5f651d4416d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:10.396568 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.396507    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-dshm" pod "1dfce1f2-1214-4926-bfeb-5f651d4416d6" (UID: "1dfce1f2-1214-4926-bfeb-5f651d4416d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.396659 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.396616    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfce1f2-1214-4926-bfeb-5f651d4416d6-kube-api-access-4k57l" pod "1dfce1f2-1214-4926-bfeb-5f651d4416d6" (UID: "1dfce1f2-1214-4926-bfeb-5f651d4416d6"). InnerVolumeSpecName "kube-api-access-4k57l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:10.497302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.497147    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dfce1f2-1214-4926-bfeb-5f651d4416d6-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.497302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.497187    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dfce1f2-1214-4926-bfeb-5f651d4416d6-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.497302 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.497202    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k57l\" (UniqueName: \"kubernetes.io/projected/1dfce1f2-1214-4926-bfeb-5f651d4416d6-kube-api-access-4k57l\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.851328 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.851288    4742 generic.go:356] "Generic (PLEG): container finished" podID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerID="ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32" exitCode=0
Oct 29 22:59:10.851500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.851362    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"
Oct 29 22:59:10.851500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.851378    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" event={"ID":"1dfce1f2-1214-4926-bfeb-5f651d4416d6","Type":"ContainerDied","Data":"ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32"}
Oct 29 22:59:10.851500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.851436    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" event={"ID":"1dfce1f2-1214-4926-bfeb-5f651d4416d6","Type":"ContainerDied","Data":"f1ba3461d0ce114e8ca910e5fc3244f31a442373cbc56dd917b534536f310887"}
Oct 29 22:59:10.851500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.851461    4742 scope.go:122] "RemoveContainer" containerID="ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32"
Oct 29 22:59:10.853682 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.853653    4742 generic.go:356] "Generic (PLEG): container finished" podID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerID="fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67" exitCode=0
Oct 29 22:59:10.853908 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.853748    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"
Oct 29 22:59:10.854877 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.853760    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" event={"ID":"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2","Type":"ContainerDied","Data":"fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67"}
Oct 29 22:59:10.854877 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.854049    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf" event={"ID":"5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2","Type":"ContainerDied","Data":"5d085f7828aa1e054c13bbe9a23bfd48d895fe8e919fe4894bbe274cab3b47a0"}
Oct 29 22:59:10.889739 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.889712    4742 scope.go:122] "RemoveContainer" containerID="ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32"
Oct 29 22:59:10.890137 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:10.890113    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32\": container with ID starting with ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32 not found: ID does not exist" containerID="ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32"
Oct 29 22:59:10.890235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.890148    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32"} err="failed to get container status \"ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32\": rpc error: code = NotFound desc = could not find container \"ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32\": container with ID starting with ddc7f3e2ac456d8fa72ac5e4957d0b6513d2b345fa6cc46fd81a68f537d8cc32 not found: ID does not exist"
Oct 29 22:59:10.890235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.890168    4742 scope.go:122] "RemoveContainer" containerID="fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67"
Oct 29 22:59:10.900444 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.900269    4742 scope.go:122] "RemoveContainer" containerID="fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67"
Oct 29 22:59:10.900715 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:10.900606    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67\": container with ID starting with fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67 not found: ID does not exist" containerID="fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67"
Oct 29 22:59:10.900715 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.900637    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67"} err="failed to get container status \"fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67\": rpc error: code = NotFound desc = could not find container \"fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67\": container with ID starting with fcd75f0256e3bb1891de5b187dbdc331a7ac53d105e40f554d7c5e5fb5d97f67 not found: ID does not exist"
Oct 29 22:59:10.956649 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.956618    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"]
Oct 29 22:59:10.971020 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.970989    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/premium-w4-dd3ff3-kserve-f7fb76964-5lswf"]
Oct 29 22:59:10.993132 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:10.993100    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" path="/var/lib/kubelet/pods/5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2/volumes"
Oct 29 22:59:11.001557 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:11.001521    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"]
Oct 29 22:59:11.007818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:11.007791    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5"]
Oct 29 22:59:11.166538 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:11.166450    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/simulator-w4-dd3ff3-kserve-95fd9cf45-x47z5" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:12.948594 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.948556    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"]
Oct 29 22:59:12.949307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949284    4742 cpu_manager.go:414] "containerMap: removing container" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main"
Oct 29 22:59:12.949307 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949308    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main"
Oct 29 22:59:12.949439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949321    4742 cpu_manager.go:414] "containerMap: removing container" podUID="d0af68dc-1260-4c55-a33d-352ba320d7b9" containerName="revoke-keys"
Oct 29 22:59:12.949439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949328    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0af68dc-1260-4c55-a33d-352ba320d7b9" containerName="revoke-keys"
Oct 29 22:59:12.949439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949338    4742 cpu_manager.go:414] "containerMap: removing container" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main"
Oct 29 22:59:12.949439 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949344    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main"
Oct 29 22:59:12.949627 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949612    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" containerName="main"
Oct 29 22:59:12.949666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949630    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0af68dc-1260-4c55-a33d-352ba320d7b9" containerName="revoke-keys"
Oct 29 22:59:12.949666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.949637    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5151b2-2b0c-4feb-9069-ac0cbbba3dc2" containerName="main"
Oct 29 22:59:12.954791 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.954773    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:12.995529 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.995495    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:59:12.995683 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:12.995571    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:13.017905 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.017864    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfce1f2-1214-4926-bfeb-5f651d4416d6" path="/var/lib/kubelet/pods/1dfce1f2-1214-4926-bfeb-5f651d4416d6/volumes"
Oct 29 22:59:13.018568 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.018418    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"]
Oct 29 22:59:13.042234 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.042199    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.042444 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.042261    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fsc\" (UniqueName: \"kubernetes.io/projected/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-kube-api-access-g6fsc\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.042546 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.042517    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.144945 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.144903    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.145150 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.144958    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fsc\" (UniqueName: \"kubernetes.io/projected/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-kube-api-access-g6fsc\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.145150 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.145107    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.145748 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.145723    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.147420 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.147396    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.184882 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.184806    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fsc\" (UniqueName: \"kubernetes.io/projected/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-kube-api-access-g6fsc\") pod \"maas-api-revoke-sub-24754ef4-wd6b6\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.292354 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.292262    4742 patch_prober.go:29] interesting pod/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:13.292512 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.292350    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:13.314438 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.314406    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:13.685406 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:59:13.685367    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bfd7a1_949d_46ff_ba4f_f1d2d831ef68.slice/crio-18edd550150574816cd2d7258a1cc42bca944f3ca4bb77aa38a4a7ce6160b49e WatchSource:0}: Error finding container 18edd550150574816cd2d7258a1cc42bca944f3ca4bb77aa38a4a7ce6160b49e: Status 404 returned error can't find the container with id 18edd550150574816cd2d7258a1cc42bca944f3ca4bb77aa38a4a7ce6160b49e
Oct 29 22:59:13.688355 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.688328    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"]
Oct 29 22:59:13.902711 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:13.902672    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6" event={"ID":"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68","Type":"ContainerStarted","Data":"18edd550150574816cd2d7258a1cc42bca944f3ca4bb77aa38a4a7ce6160b49e"}
Oct 29 22:59:14.139867 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.139833    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"]
Oct 29 22:59:14.144170 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.144149    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.189133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.185995    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"]
Oct 29 22:59:14.292032 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.291985    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.292246 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.292155    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/227eaa41-be04-40a5-9b0a-621acb2b96c4-kube-api-access-j2nsq\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.292437 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.292411    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.394218 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.394077    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/227eaa41-be04-40a5-9b0a-621acb2b96c4-kube-api-access-j2nsq\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.394405 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.394314    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.394405 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.394380    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.395031 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.395006    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.396760 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.396729    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.414305 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.414270    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/227eaa41-be04-40a5-9b0a-621acb2b96c4-kube-api-access-j2nsq\") pod \"maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.501944 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.501911    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:14.845239 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.845209    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"]
Oct 29 22:59:14.914208 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.914177    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4" event={"ID":"227eaa41-be04-40a5-9b0a-621acb2b96c4","Type":"ContainerStarted","Data":"3516393acf4368a3e69cdbfaeb9fcacc3dc13d7194cbc77eb46baa8c86b26202"}
Oct 29 22:59:14.915934 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.915914    4742 generic.go:356] "Generic (PLEG): container finished" podID="22bfd7a1-949d-46ff-ba4f-f1d2d831ef68" containerID="016be460ff311bc344c29bc4222e9383019bdbb95918f667afff9ff37bbf2096" exitCode=0
Oct 29 22:59:14.916005 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:14.915985    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6" event={"ID":"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68","Type":"ContainerDied","Data":"016be460ff311bc344c29bc4222e9383019bdbb95918f667afff9ff37bbf2096"}
Oct 29 22:59:15.927592 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:15.927546    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4" event={"ID":"227eaa41-be04-40a5-9b0a-621acb2b96c4","Type":"ContainerStarted","Data":"e99d53cf0663fee88172c2227ca7bc0f929abefcc1993e6ecee36a3e4ccfee75"}
Oct 29 22:59:15.930366 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:15.930333    4742 generic.go:356] "Generic (PLEG): container finished" podID="634172d1-c4be-49b8-993d-7fcbc9792698" containerID="6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d" exitCode=0
Oct 29 22:59:15.930521 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:15.930403    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" event={"ID":"634172d1-c4be-49b8-993d-7fcbc9792698","Type":"ContainerDied","Data":"6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d"}
Oct 29 22:59:15.930800 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:15.930782    4742 scope.go:122] "RemoveContainer" containerID="6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d"
Oct 29 22:59:15.932430 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:15.932396    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:59:15.932518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:15.932488    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:59:15.932630 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:15.932538    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:59:16.055674 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.055612    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4" podStartSLOduration=2.055593481 podStartE2EDuration="2.055593481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:15.9830453 +0000 UTC m=+1013.123786885" watchObservedRunningTime="2026-10-29 22:59:16.055593481 +0000 UTC m=+1013.196335067"
Oct 29 22:59:16.057076 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.057051    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"]
Oct 29 22:59:16.070314 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.062429    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n" podUID="06e35b87-8978-47a2-b4d5-423f76d30f21" containerName="istio-proxy" containerID="cri-o://9b18a87c0c22a5e294c798cf813287ba3bf4e140278d35c558ad02576cf7579e" gracePeriod=30
Oct 29 22:59:16.518325 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.518283    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:16.529397 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.529358    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-xks-ca\") pod \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") "
Oct 29 22:59:16.529601 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.529446    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-kube-api-access-g6fsc\" (UniqueName: \"kubernetes.io/projected/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-kube-api-access-g6fsc\") pod \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") "
Oct 29 22:59:16.529601 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.529487    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-openshift-ca\") pod \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\" (UID: \"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68\") "
Oct 29 22:59:16.531554 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.531506    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-openshift-ca" pod "22bfd7a1-949d-46ff-ba4f-f1d2d831ef68" (UID: "22bfd7a1-949d-46ff-ba4f-f1d2d831ef68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:16.533657 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.533624    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-kube-api-access-g6fsc" pod "22bfd7a1-949d-46ff-ba4f-f1d2d831ef68" (UID: "22bfd7a1-949d-46ff-ba4f-f1d2d831ef68"). InnerVolumeSpecName "kube-api-access-g6fsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:16.533966 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.533946    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-xks-ca" pod "22bfd7a1-949d-46ff-ba4f-f1d2d831ef68" (UID: "22bfd7a1-949d-46ff-ba4f-f1d2d831ef68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:16.631606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.631571    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.631606 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.631608    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6fsc\" (UniqueName: \"kubernetes.io/projected/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-kube-api-access-g6fsc\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.631822 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.631623    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.943878 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.943847    4742 generic.go:356] "Generic (PLEG): container finished" podID="227eaa41-be04-40a5-9b0a-621acb2b96c4" containerID="e99d53cf0663fee88172c2227ca7bc0f929abefcc1993e6ecee36a3e4ccfee75" exitCode=0
Oct 29 22:59:16.944347 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.943933    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4" event={"ID":"227eaa41-be04-40a5-9b0a-621acb2b96c4","Type":"ContainerDied","Data":"e99d53cf0663fee88172c2227ca7bc0f929abefcc1993e6ecee36a3e4ccfee75"}
Oct 29 22:59:16.946311 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.946285    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" event={"ID":"634172d1-c4be-49b8-993d-7fcbc9792698","Type":"ContainerStarted","Data":"b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2"}
Oct 29 22:59:16.947821 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.947804    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"
Oct 29 22:59:16.947925 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.947880    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6" event={"ID":"22bfd7a1-949d-46ff-ba4f-f1d2d831ef68","Type":"ContainerDied","Data":"18edd550150574816cd2d7258a1cc42bca944f3ca4bb77aa38a4a7ce6160b49e"}
Oct 29 22:59:16.947979 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.947923    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18edd550150574816cd2d7258a1cc42bca944f3ca4bb77aa38a4a7ce6160b49e"
Oct 29 22:59:16.996304 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:16.974478    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:59:17.994822 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:17.994771    4742 patch_prober.go:29] interesting pod/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:59:17.995461 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:17.995414    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 29 22:59:18.236532 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.236504    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:18.349473 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.349435    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227eaa41-be04-40a5-9b0a-621acb2b96c4-kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/227eaa41-be04-40a5-9b0a-621acb2b96c4-kube-api-access-j2nsq\") pod \"227eaa41-be04-40a5-9b0a-621acb2b96c4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") "
Oct 29 22:59:18.349670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.349506    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-openshift-ca\") pod \"227eaa41-be04-40a5-9b0a-621acb2b96c4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") "
Oct 29 22:59:18.349670 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.349604    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-xks-ca\") pod \"227eaa41-be04-40a5-9b0a-621acb2b96c4\" (UID: \"227eaa41-be04-40a5-9b0a-621acb2b96c4\") "
Oct 29 22:59:18.349923 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.349895    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-openshift-ca" pod "227eaa41-be04-40a5-9b0a-621acb2b96c4" (UID: "227eaa41-be04-40a5-9b0a-621acb2b96c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:18.350213 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.350188    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.351853 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.351820    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227eaa41-be04-40a5-9b0a-621acb2b96c4-kube-api-access-j2nsq" pod "227eaa41-be04-40a5-9b0a-621acb2b96c4" (UID: "227eaa41-be04-40a5-9b0a-621acb2b96c4"). InnerVolumeSpecName "kube-api-access-j2nsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:18.352205 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.352184    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-xks-ca" pod "227eaa41-be04-40a5-9b0a-621acb2b96c4" (UID: "227eaa41-be04-40a5-9b0a-621acb2b96c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:18.452198 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.452160    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2nsq\" (UniqueName: \"kubernetes.io/projected/227eaa41-be04-40a5-9b0a-621acb2b96c4-kube-api-access-j2nsq\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.452198 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.452200    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227eaa41-be04-40a5-9b0a-621acb2b96c4-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.981098 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.979487    4742 patch_prober.go:29] interesting pod/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:59:18.981098 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:18.979547    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 29 22:59:19.005401 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:19.005371    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4" event={"ID":"227eaa41-be04-40a5-9b0a-621acb2b96c4","Type":"ContainerDied","Data":"3516393acf4368a3e69cdbfaeb9fcacc3dc13d7194cbc77eb46baa8c86b26202"}
Oct 29 22:59:19.005824 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:19.005417    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3516393acf4368a3e69cdbfaeb9fcacc3dc13d7194cbc77eb46baa8c86b26202"
Oct 29 22:59:19.005824 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:19.005425    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"
Oct 29 22:59:19.891253 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:19.891206    4742 patch_prober.go:29] interesting pod/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:19.891417 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:19.891269    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n" podUID="06e35b87-8978-47a2-b4d5-423f76d30f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:19.973959 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:19.973506    4742 patch_prober.go:29] interesting pod/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:59:19.973959 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:19.973572    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 29 22:59:20.232074 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:20.231986    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf" podUID="c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:20.232074 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:20.232060    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798" podUID="5620b0a7-c9a8-4652-b202-d522ac93f01b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:20.980109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:20.979489    4742 patch_prober.go:29] interesting pod/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:59:20.980109 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:20.979570    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 29 22:59:21.980108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:21.973501    4742 patch_prober.go:29] interesting pod/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:59:21.980108 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:21.973576    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 29 22:59:22.833368 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.833317    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"]
Oct 29 22:59:22.835625 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.834267    4742 cpu_manager.go:414] "containerMap: removing container" podUID="22bfd7a1-949d-46ff-ba4f-f1d2d831ef68" containerName="revoke-keys"
Oct 29 22:59:22.835625 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.834287    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22bfd7a1-949d-46ff-ba4f-f1d2d831ef68" containerName="revoke-keys"
Oct 29 22:59:22.835625 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.834346    4742 cpu_manager.go:414] "containerMap: removing container" podUID="227eaa41-be04-40a5-9b0a-621acb2b96c4" containerName="revoke-keys"
Oct 29 22:59:22.835625 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.834355    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227eaa41-be04-40a5-9b0a-621acb2b96c4" containerName="revoke-keys"
Oct 29 22:59:22.835625 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.834482    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="227eaa41-be04-40a5-9b0a-621acb2b96c4" containerName="revoke-keys"
Oct 29 22:59:22.835625 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.834499    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="22bfd7a1-949d-46ff-ba4f-f1d2d831ef68" containerName="revoke-keys"
Oct 29 22:59:22.838813 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.838790    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:22.856464 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.856433    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"]
Oct 29 22:59:22.857455 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.857428    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg57d\" (UniqueName: \"kubernetes.io/projected/ff7d5521-a843-4354-bbb0-c3d042da8f4e-kube-api-access-gg57d\") pod \"payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw\" (UID: \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:22.857583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.857477    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7d5521-a843-4354-bbb0-c3d042da8f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw\" (UID: \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:22.859444 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.859411    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-34d0fd42\""
Oct 29 22:59:22.865289 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.865260    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-34d0fd42-dockercfg-ktxlp\""
Oct 29 22:59:22.972056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.971304    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg57d\" (UniqueName: \"kubernetes.io/projected/ff7d5521-a843-4354-bbb0-c3d042da8f4e-kube-api-access-gg57d\") pod \"payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw\" (UID: \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:22.972056 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:22.971358    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7d5521-a843-4354-bbb0-c3d042da8f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw\" (UID: \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:23.015299 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.011879    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-34d0fd42\""
Oct 29 22:59:23.015832 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.015311    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg57d\" (UniqueName: \"kubernetes.io/projected/ff7d5521-a843-4354-bbb0-c3d042da8f4e-kube-api-access-gg57d\") pod \"payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw\" (UID: \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:23.023249 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.022441    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7d5521-a843-4354-bbb0-c3d042da8f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw\" (UID: \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:23.129197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.129171    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:59:23.129859 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.129803    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:59:23.132003 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.131978    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:59:23.149814 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.149775    4742 generic.go:356] "Generic (PLEG): container finished" podID="06e35b87-8978-47a2-b4d5-423f76d30f21" containerID="9b18a87c0c22a5e294c798cf813287ba3bf4e140278d35c558ad02576cf7579e" exitCode=0
Oct 29 22:59:23.150001 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.149976    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n" event={"ID":"06e35b87-8978-47a2-b4d5-423f76d30f21","Type":"ContainerDied","Data":"9b18a87c0c22a5e294c798cf813287ba3bf4e140278d35c558ad02576cf7579e"}
Oct 29 22:59:23.166278 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:23.166248    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:59:23.175428 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.175398    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-34d0fd42-dockercfg-ktxlp\""
Oct 29 22:59:23.182500 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.182475    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:23.335265 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.334886    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:59:23.414197 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414077    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-data\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414377 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414203    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-kube-api-access-xdcmb\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-kube-api-access-xdcmb\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414377 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414255    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-podinfo\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414377 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414316    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-socket\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414377 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414341    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-token\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-token\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414609 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414390    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-credential-socket\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414609 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414425    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-data" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:23.414609 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414439    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e35b87-8978-47a2-b4d5-423f76d30f21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e35b87-8978-47a2-b4d5-423f76d30f21-istiod-ca-cert\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414767 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414629    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-certs\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414767 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414661    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-envoy\") pod \"06e35b87-8978-47a2-b4d5-423f76d30f21\" (UID: \"06e35b87-8978-47a2-b4d5-423f76d30f21\") "
Oct 29 22:59:23.414767 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414704    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-socket" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:23.414940 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414920    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-certs" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:23.414996 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.414948    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e35b87-8978-47a2-b4d5-423f76d30f21-istiod-ca-cert" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:23.415454 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.415425    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.415454 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.415453    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e35b87-8978-47a2-b4d5-423f76d30f21-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.415652 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.415468    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.415652 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.415481    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.415768 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.415740    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-credential-socket" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:23.417049 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.417018    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-podinfo" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:23.417194 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.417161    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-kube-api-access-xdcmb" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "kube-api-access-xdcmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:23.417358 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.417335    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-envoy" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:23.417860 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.417837    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-token" pod "06e35b87-8978-47a2-b4d5-423f76d30f21" (UID: "06e35b87-8978-47a2-b4d5-423f76d30f21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:23.517274 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.517238    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdcmb\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-kube-api-access-xdcmb\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.517274 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.517271    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.517274 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.517280    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.517274 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.517289    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.517581 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.517298    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e35b87-8978-47a2-b4d5-423f76d30f21-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:23.554523 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:59:23.554489    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7d5521_a843_4354_bbb0_c3d042da8f4e.slice/crio-7bef9650e736e04ff4aa650f4e116b0ba5f49905c87f5ebf67c87236889ecf83 WatchSource:0}: Error finding container 7bef9650e736e04ff4aa650f4e116b0ba5f49905c87f5ebf67c87236889ecf83: Status 404 returned error can't find the container with id 7bef9650e736e04ff4aa650f4e116b0ba5f49905c87f5ebf67c87236889ecf83
Oct 29 22:59:23.555799 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:23.555779    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"]
Oct 29 22:59:24.161301 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.161242    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw" event={"ID":"ff7d5521-a843-4354-bbb0-c3d042da8f4e","Type":"ContainerStarted","Data":"5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2"}
Oct 29 22:59:24.161301 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.161318    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw" event={"ID":"ff7d5521-a843-4354-bbb0-c3d042da8f4e","Type":"ContainerStarted","Data":"7bef9650e736e04ff4aa650f4e116b0ba5f49905c87f5ebf67c87236889ecf83"}
Oct 29 22:59:24.161883 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.161365    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:24.163333 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.163310    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"
Oct 29 22:59:24.163460 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.163357    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n" event={"ID":"06e35b87-8978-47a2-b4d5-423f76d30f21","Type":"ContainerDied","Data":"eae189186b242bba0031985930a10769a520ca5023275a6dd65b2bf4cc4b4441"}
Oct 29 22:59:24.163460 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.163417    4742 scope.go:122] "RemoveContainer" containerID="9b18a87c0c22a5e294c798cf813287ba3bf4e140278d35c558ad02576cf7579e"
Oct 29 22:59:24.202914 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.202865    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw" podStartSLOduration=2.202851056 podStartE2EDuration="2.202851056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:24.200516291 +0000 UTC m=+1021.341257875" watchObservedRunningTime="2026-10-29 22:59:24.202851056 +0000 UTC m=+1021.343592639"
Oct 29 22:59:24.242355 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.242328    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"]
Oct 29 22:59:24.248064 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.248032    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2bd3f07e-openshift-default-b864b98bc-gjq2n"]
Oct 29 22:59:24.995562 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:24.995529    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e35b87-8978-47a2-b4d5-423f76d30f21" path="/var/lib/kubelet/pods/06e35b87-8978-47a2-b4d5-423f76d30f21/volumes"
Oct 29 22:59:25.747423 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:25.747400    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:59:25.877563 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:25.877531    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-kube-api-access-7lbvn\" (UniqueName: \"kubernetes.io/projected/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-kube-api-access-7lbvn\") pod \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\" (UID: \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\") "
Oct 29 22:59:25.877804 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:25.877640    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-plugins-config-volume\") pod \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\" (UID: \"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26\") "
Oct 29 22:59:25.877989 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:25.877962    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-plugins-config-volume" pod "c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" (UID: "c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:25.878177 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:25.878163    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:25.879636 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:25.879616    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-kube-api-access-7lbvn" pod "c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" (UID: "c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26"). InnerVolumeSpecName "kube-api-access-7lbvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:25.979396 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:25.979302    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lbvn\" (UniqueName: \"kubernetes.io/projected/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26-kube-api-access-7lbvn\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:26.180802 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.180757    4742 generic.go:356] "Generic (PLEG): container finished" podID="c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" containerID="676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee" exitCode=0
Oct 29 22:59:26.180962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.180812    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"
Oct 29 22:59:26.180962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.180837    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf" event={"ID":"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26","Type":"ContainerDied","Data":"676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee"}
Oct 29 22:59:26.180962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.180891    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf" event={"ID":"c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26","Type":"ContainerDied","Data":"34b27d0ac94a264a4f43bd9b5dac76ff33e0161007559b1f2c1811a99db54354"}
Oct 29 22:59:26.180962 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.180912    4742 scope.go:122] "RemoveContainer" containerID="676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee"
Oct 29 22:59:26.206260 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.206231    4742 scope.go:122] "RemoveContainer" containerID="676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee"
Oct 29 22:59:26.206542 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:26.206525    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee\": container with ID starting with 676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee not found: ID does not exist" containerID="676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee"
Oct 29 22:59:26.206592 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.206551    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee"} err="failed to get container status \"676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee\": rpc error: code = NotFound desc = could not find container \"676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee\": container with ID starting with 676288bc5b974b1938791183dc73e133690c2cd463a891e1183c40da305b8bee not found: ID does not exist"
Oct 29 22:59:26.248406 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.248306    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"]
Oct 29 22:59:26.252867 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.252837    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dd3ff3-ff9b665d9-b5ppf"]
Oct 29 22:59:26.360034 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.360004    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:59:26.486173 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.486135    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5620b0a7-c9a8-4652-b202-d522ac93f01b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5620b0a7-c9a8-4652-b202-d522ac93f01b-plugins-config-volume\") pod \"5620b0a7-c9a8-4652-b202-d522ac93f01b\" (UID: \"5620b0a7-c9a8-4652-b202-d522ac93f01b\") "
Oct 29 22:59:26.486505 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.486353    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5620b0a7-c9a8-4652-b202-d522ac93f01b-kube-api-access-796sf\" (UniqueName: \"kubernetes.io/projected/5620b0a7-c9a8-4652-b202-d522ac93f01b-kube-api-access-796sf\") pod \"5620b0a7-c9a8-4652-b202-d522ac93f01b\" (UID: \"5620b0a7-c9a8-4652-b202-d522ac93f01b\") "
Oct 29 22:59:26.486688 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.486655    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5620b0a7-c9a8-4652-b202-d522ac93f01b-plugins-config-volume" pod "5620b0a7-c9a8-4652-b202-d522ac93f01b" (UID: "5620b0a7-c9a8-4652-b202-d522ac93f01b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:26.486977 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.486955    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5620b0a7-c9a8-4652-b202-d522ac93f01b-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:26.488729 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.488707    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5620b0a7-c9a8-4652-b202-d522ac93f01b-kube-api-access-796sf" pod "5620b0a7-c9a8-4652-b202-d522ac93f01b" (UID: "5620b0a7-c9a8-4652-b202-d522ac93f01b"). InnerVolumeSpecName "kube-api-access-796sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:26.587907 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.587871    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796sf\" (UniqueName: \"kubernetes.io/projected/5620b0a7-c9a8-4652-b202-d522ac93f01b-kube-api-access-796sf\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:26.992191 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:26.992156    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" path="/var/lib/kubelet/pods/c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26/volumes"
Oct 29 22:59:27.190036 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.190003    4742 generic.go:356] "Generic (PLEG): container finished" podID="5620b0a7-c9a8-4652-b202-d522ac93f01b" containerID="fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207" exitCode=0
Oct 29 22:59:27.190222 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.190049    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"
Oct 29 22:59:27.190222 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.190096    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798" event={"ID":"5620b0a7-c9a8-4652-b202-d522ac93f01b","Type":"ContainerDied","Data":"fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207"}
Oct 29 22:59:27.190222 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.190150    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798" event={"ID":"5620b0a7-c9a8-4652-b202-d522ac93f01b","Type":"ContainerDied","Data":"d2328eb2c29ae613e3323b194b2b297cd6f424303b9691b3d6c40b358900b204"}
Oct 29 22:59:27.190222 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.190173    4742 scope.go:122] "RemoveContainer" containerID="fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207"
Oct 29 22:59:27.199949 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.199927    4742 scope.go:122] "RemoveContainer" containerID="fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207"
Oct 29 22:59:27.200245 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:27.200223    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207\": container with ID starting with fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207 not found: ID does not exist" containerID="fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207"
Oct 29 22:59:27.200311 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.200253    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207"} err="failed to get container status \"fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207\": rpc error: code = NotFound desc = could not find container \"fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207\": container with ID starting with fd22b79eb1d80043ec9aafe424369aaeaf558f306443fb16988f1c6327e83207 not found: ID does not exist"
Oct 29 22:59:27.247477 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.247384    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"]
Oct 29 22:59:27.259121 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:27.259094    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dd3ff3-6d87995487-np798"]
Oct 29 22:59:28.992185 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:28.992154    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5620b0a7-c9a8-4652-b202-d522ac93f01b" path="/var/lib/kubelet/pods/5620b0a7-c9a8-4652-b202-d522ac93f01b/volumes"
Oct 29 22:59:32.045364 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:32.045318    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"]
Oct 29 22:59:32.056725 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:32.056695    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f50282-wxjpm"]
Oct 29 22:59:32.993783 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:32.993750    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b603f0e6-fc00-410d-9a9b-bd186e3ff01c" path="/var/lib/kubelet/pods/b603f0e6-fc00-410d-9a9b-bd186e3ff01c/volumes"
Oct 29 22:59:33.025206 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:33.025173    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"]
Oct 29 22:59:33.037122 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:33.037077    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c373e5a5-q7p9q"]
Oct 29 22:59:34.799445 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.799411    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"]
Oct 29 22:59:34.800037 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800021    4742 cpu_manager.go:414] "containerMap: removing container" podUID="5620b0a7-c9a8-4652-b202-d522ac93f01b" containerName="payload-pre-processing"
Oct 29 22:59:34.800128 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800038    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5620b0a7-c9a8-4652-b202-d522ac93f01b" containerName="payload-pre-processing"
Oct 29 22:59:34.800128 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800052    4742 cpu_manager.go:414] "containerMap: removing container" podUID="06e35b87-8978-47a2-b4d5-423f76d30f21" containerName="istio-proxy"
Oct 29 22:59:34.800128 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800058    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e35b87-8978-47a2-b4d5-423f76d30f21" containerName="istio-proxy"
Oct 29 22:59:34.800128 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800096    4742 cpu_manager.go:414] "containerMap: removing container" podUID="c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" containerName="payload-processing"
Oct 29 22:59:34.800128 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800104    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" containerName="payload-processing"
Oct 29 22:59:34.800284 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800211    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="5620b0a7-c9a8-4652-b202-d522ac93f01b" containerName="payload-pre-processing"
Oct 29 22:59:34.800284 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800219    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e35b87-8978-47a2-b4d5-423f76d30f21" containerName="istio-proxy"
Oct 29 22:59:34.800284 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.800229    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e6398e-bbf3-4fe6-a5e5-f6e617a89f26" containerName="payload-processing"
Oct 29 22:59:34.805416 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.805375    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.823174 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.822242    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-34d0fd42-serving-cert\""
Oct 29 22:59:34.829434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.829408    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"]
Oct 29 22:59:34.833684 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.833652    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.833818 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.833716    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-metrics-certs\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.833879 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.833819    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13562bb-15c0-478d-9a25-2644bf11c412-openshift-service-ca\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.833879 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.833869    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7hw\" (UniqueName: \"kubernetes.io/projected/e13562bb-15c0-478d-9a25-2644bf11c412-kube-api-access-dc7hw\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.935349 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.935313    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.935520 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.935370    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-metrics-certs\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.935520 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.935452    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13562bb-15c0-478d-9a25-2644bf11c412-openshift-service-ca\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.935520 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:34.935482    4742 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-serving-cert: secret "maas-api-e2e-ait-34d0fd42-serving-cert" not found
Oct 29 22:59:34.935659 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.935564    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7hw\" (UniqueName: \"kubernetes.io/projected/e13562bb-15c0-478d-9a25-2644bf11c412-kube-api-access-dc7hw\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.935659 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:34.935583    4742 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls podName:e13562bb-15c0-478d-9a25-2644bf11c412 nodeName:}" failed. No retries permitted until 2026-10-29 22:59:35.435562154 +0000 UTC m=+1032.576303736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls") pod "maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d" (UID: "e13562bb-15c0-478d-9a25-2644bf11c412") : secret "maas-api-e2e-ait-34d0fd42-serving-cert" not found
Oct 29 22:59:34.936360 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.936340    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13562bb-15c0-478d-9a25-2644bf11c412-openshift-service-ca\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.937853 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.937832    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-metrics-certs\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.956243 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.956216    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7hw\" (UniqueName: \"kubernetes.io/projected/e13562bb-15c0-478d-9a25-2644bf11c412-kube-api-access-dc7hw\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:34.993835 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:34.993804    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26fda6b-df94-40cd-9b24-d6aad8d90390" path="/var/lib/kubelet/pods/c26fda6b-df94-40cd-9b24-d6aad8d90390/volumes"
Oct 29 22:59:35.206377 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:35.206343    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 22:59:35.485887 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:35.485795    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:35.488541 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:35.488513    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls\") pod \"maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:35.727908 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:35.727816    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:36.145416 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:36.135560    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"]
Oct 29 22:59:36.441296 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:36.422451    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d" event={"ID":"e13562bb-15c0-478d-9a25-2644bf11c412","Type":"ContainerStarted","Data":"9d37908f933d7e76b8e84491f074d6f3d8db958ead132ff0850f0399a5dc133e"}
Oct 29 22:59:36.885463 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:36.883801    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"]
Oct 29 22:59:36.886107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:36.885689    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy" containerID="cri-o://338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa" gracePeriod=30
Oct 29 22:59:36.893235 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:36.893205    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"]
Oct 29 22:59:36.893520 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:36.893492    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy" containerID="cri-o://87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe" gracePeriod=30
Oct 29 22:59:36.898668 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:36.898644    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"]
Oct 29 22:59:36.899070 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:36.898870    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy" containerID="cri-o://199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116" gracePeriod=30
Oct 29 22:59:37.431158 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:37.431117    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d" event={"ID":"e13562bb-15c0-478d-9a25-2644bf11c412","Type":"ContainerStarted","Data":"75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2"}
Oct 29 22:59:37.431556 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:37.431187    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:37.500800 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:37.500745    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d" podStartSLOduration=3.02561385 podStartE2EDuration="3.500731301s" totalImagesPullingTime="475.117451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:37.499504279 +0000 UTC m=+1034.640245865" watchObservedRunningTime="2026-10-29 22:59:37.500731301 +0000 UTC m=+1034.641472884"
Oct 29 22:59:40.002457 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.002418    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"]
Oct 29 22:59:40.002991 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.002718    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5" podUID="f165aa12-7e76-424b-b4ec-02b808b8b281" containerName="maas-api" containerID="cri-o://7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5" gracePeriod=30
Oct 29 22:59:40.320865 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.320836    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:59:40.417696 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.417658    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f165aa12-7e76-424b-b4ec-02b808b8b281-kube-api-access-4jtrk\" (UniqueName: \"kubernetes.io/projected/f165aa12-7e76-424b-b4ec-02b808b8b281-kube-api-access-4jtrk\") pod \"f165aa12-7e76-424b-b4ec-02b808b8b281\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") "
Oct 29 22:59:40.417894 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.417873    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-metrics-certs\") pod \"f165aa12-7e76-424b-b4ec-02b808b8b281\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") "
Oct 29 22:59:40.418129 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.418061    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f165aa12-7e76-424b-b4ec-02b808b8b281-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f165aa12-7e76-424b-b4ec-02b808b8b281-openshift-service-ca\") pod \"f165aa12-7e76-424b-b4ec-02b808b8b281\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") "
Oct 29 22:59:40.418222 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.418131    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-maas-api-tls\") pod \"f165aa12-7e76-424b-b4ec-02b808b8b281\" (UID: \"f165aa12-7e76-424b-b4ec-02b808b8b281\") "
Oct 29 22:59:40.419579 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.419509    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f165aa12-7e76-424b-b4ec-02b808b8b281-openshift-service-ca" pod "f165aa12-7e76-424b-b4ec-02b808b8b281" (UID: "f165aa12-7e76-424b-b4ec-02b808b8b281"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:40.419866 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.419840    4742 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f165aa12-7e76-424b-b4ec-02b808b8b281-openshift-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.421002 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.420970    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-metrics-certs" pod "f165aa12-7e76-424b-b4ec-02b808b8b281" (UID: "f165aa12-7e76-424b-b4ec-02b808b8b281"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:40.421315 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.421293    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-maas-api-tls" pod "f165aa12-7e76-424b-b4ec-02b808b8b281" (UID: "f165aa12-7e76-424b-b4ec-02b808b8b281"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:40.421993 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.421973    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f165aa12-7e76-424b-b4ec-02b808b8b281-kube-api-access-4jtrk" pod "f165aa12-7e76-424b-b4ec-02b808b8b281" (UID: "f165aa12-7e76-424b-b4ec-02b808b8b281"). InnerVolumeSpecName "kube-api-access-4jtrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:40.459735 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.459700    4742 generic.go:356] "Generic (PLEG): container finished" podID="f165aa12-7e76-424b-b4ec-02b808b8b281" containerID="7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5" exitCode=0
Oct 29 22:59:40.459914 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.459775    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"
Oct 29 22:59:40.459914 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.459764    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5" event={"ID":"f165aa12-7e76-424b-b4ec-02b808b8b281","Type":"ContainerDied","Data":"7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5"}
Oct 29 22:59:40.460158 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.460135    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5" event={"ID":"f165aa12-7e76-424b-b4ec-02b808b8b281","Type":"ContainerDied","Data":"df8f4d59da00a52e6a3251ad0cad2523f46eb57b836c24d57a9547831aa97888"}
Oct 29 22:59:40.460228 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.460185    4742 scope.go:122] "RemoveContainer" containerID="7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5"
Oct 29 22:59:40.471827 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.471806    4742 scope.go:122] "RemoveContainer" containerID="7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5"
Oct 29 22:59:40.472117 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:40.472080    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5\": container with ID starting with 7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5 not found: ID does not exist" containerID="7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5"
Oct 29 22:59:40.472202 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.472124    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5"} err="failed to get container status \"7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5\": rpc error: code = NotFound desc = could not find container \"7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5\": container with ID starting with 7f0d38079b50d8465a1bfed496ec411a1fbe858a4487da8612028ff9ea0991e5 not found: ID does not exist"
Oct 29 22:59:40.513013 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.512981    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"]
Oct 29 22:59:40.519971 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.519935    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5"]
Oct 29 22:59:40.520788 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.520770    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-maas-api-tls\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.520857 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.520792    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jtrk\" (UniqueName: \"kubernetes.io/projected/f165aa12-7e76-424b-b4ec-02b808b8b281-kube-api-access-4jtrk\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.520857 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.520813    4742 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f165aa12-7e76-424b-b4ec-02b808b8b281-metrics-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.845268 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.845237    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"]
Oct 29 22:59:40.849623 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:40.849596    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-01e33b3d-snqj4"]
Oct 29 22:59:41.018403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:41.018367    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227eaa41-be04-40a5-9b0a-621acb2b96c4" path="/var/lib/kubelet/pods/227eaa41-be04-40a5-9b0a-621acb2b96c4/volumes"
Oct 29 22:59:41.020061 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:41.020033    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f165aa12-7e76-424b-b4ec-02b808b8b281" path="/var/lib/kubelet/pods/f165aa12-7e76-424b-b4ec-02b808b8b281/volumes"
Oct 29 22:59:41.020521 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:41.020490    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"]
Oct 29 22:59:41.020781 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:41.020756    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy" containerID="cri-o://b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2" gracePeriod=30
Oct 29 22:59:42.099328 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.098560    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"]
Oct 29 22:59:42.100271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.099591    4742 cpu_manager.go:414] "containerMap: removing container" podUID="f165aa12-7e76-424b-b4ec-02b808b8b281" containerName="maas-api"
Oct 29 22:59:42.100271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.099613    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f165aa12-7e76-424b-b4ec-02b808b8b281" containerName="maas-api"
Oct 29 22:59:42.100271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.099907    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="f165aa12-7e76-424b-b4ec-02b808b8b281" containerName="maas-api"
Oct 29 22:59:42.105468 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.105422    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.128639 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.127979    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-28ed1cea-openshift-default-dockercfg-8wbk6\""
Oct 29 22:59:42.128639 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.128122    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"]
Oct 29 22:59:42.233424 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.232186    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:59:42.234647 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.234611    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:59:42.250628 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250567    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-token\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.251294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250631    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-data\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.251294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250707    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-envoy\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.251294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250733    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istiod-ca-cert\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.251294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250774    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-podinfo\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.251294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250802    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-certs\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.251294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250852    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-socket\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.251294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250893    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6j4t\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-kube-api-access-h6j4t\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.251294 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.250996    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-credential-socket\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.325987 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.325956    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:59:42.351542 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351510    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-certs\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.351542 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351557    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-socket\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.351877 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351615    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef3ee10-d844-498e-be25-9f151d022caa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef3ee10-d844-498e-be25-9f151d022caa-istiod-ca-cert\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.351877 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351684    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-credential-socket\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.351877 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351722    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-envoy\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.351877 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351761    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-credential-socket\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.351877 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351817    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-certs\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.352252 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351892    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-data\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.352252 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.351974    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-istio-token\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-istio-token\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.352252 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.352094    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-data\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.352252 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.352136    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-istio-token\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-istio-token\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.352252 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.352173    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-kube-api-access-8wvzg\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-kube-api-access-8wvzg\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.352252 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.352210    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-envoy\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.352664 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.352469    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-socket" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.352749 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.352664    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-credential-socket" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.353312 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.353176    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef3ee10-d844-498e-be25-9f151d022caa-istiod-ca-cert" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:42.354671 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.354583    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-envoy" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.354974 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.354952    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-certs" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.355259 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.355164    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-data" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.357730 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.357703    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-kube-api-access-8wvzg" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "kube-api-access-8wvzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:42.357827 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.357795    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-socket\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.357888 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.357848    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bef3ee10-d844-498e-be25-9f151d022caa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef3ee10-d844-498e-be25-9f151d022caa-istio-podinfo\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.357939 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.357908    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a71d461-c056-47b8-ae80-a12994784a4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a71d461-c056-47b8-ae80-a12994784a4e-istiod-ca-cert\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.357997 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.357962    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a71d461-c056-47b8-ae80-a12994784a4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a71d461-c056-47b8-ae80-a12994784a4e-istio-podinfo\") pod \"6a71d461-c056-47b8-ae80-a12994784a4e\" (UID: \"6a71d461-c056-47b8-ae80-a12994784a4e\") "
Oct 29 22:59:42.358049 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.358008    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-kube-api-access-wgd5d\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-kube-api-access-wgd5d\") pod \"bef3ee10-d844-498e-be25-9f151d022caa\" (UID: \"bef3ee10-d844-498e-be25-9f151d022caa\") "
Oct 29 22:59:42.358362 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.358326    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-envoy\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.358537 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.358517    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istiod-ca-cert\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.358822 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.358662    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-podinfo\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.358971 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.358952    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-certs\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.359156 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.359131    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-socket\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.359309 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.359290    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6j4t\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-kube-api-access-h6j4t\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.359523 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.359506    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-credential-socket\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.360263 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.359867    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-token\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.360464 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.360444    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-data\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.360688 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.360674    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef3ee10-d844-498e-be25-9f151d022caa-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.360790 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.360779    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.360884 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.360870    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.360969 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.360957    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.361044 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.361034    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.361143 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.361131    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wvzg\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-kube-api-access-8wvzg\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.361284 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.361272    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.361691 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.361673    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-data\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.362458 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.362431    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-certs\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.362921 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.362895    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istiod-ca-cert\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.363665 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.363643    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-socket\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.364518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.358733    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-data" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.364518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.360113    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-credential-socket" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.364518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.362251    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-envoy" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.364518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.363067    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-certs" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.364518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.363775    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a71d461-c056-47b8-ae80-a12994784a4e-istio-podinfo" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:42.364518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.364218    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-socket" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.364518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.364275    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a71d461-c056-47b8-ae80-a12994784a4e-istiod-ca-cert" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:42.364518 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.364478    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-credential-socket\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.367508 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.367484    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-istio-token" pod "6a71d461-c056-47b8-ae80-a12994784a4e" (UID: "6a71d461-c056-47b8-ae80-a12994784a4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:42.368125 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.368066    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-podinfo\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.368373 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.368316    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-istio-token" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:42.368666 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.368639    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-kube-api-access-wgd5d" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "kube-api-access-wgd5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:42.369312 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.369288    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-envoy\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.370400 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.370364    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bef3ee10-d844-498e-be25-9f151d022caa-istio-podinfo" pod "bef3ee10-d844-498e-be25-9f151d022caa" (UID: "bef3ee10-d844-498e-be25-9f151d022caa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:42.388133 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.388061    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6j4t\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-kube-api-access-h6j4t\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.392580 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.392550    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-token\") pod \"e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") " pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.429817 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.429498    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.462602    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-data\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.462660    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-envoy\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.462836    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-podinfo\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.462880    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-token\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-token\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.462992    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-certs\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463027    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-kube-api-access-6chzg\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-kube-api-access-6chzg\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463105    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-socket\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463167    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-credential-socket\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463194    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcca471e-1edf-4fbe-9802-7d83169bf764-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcca471e-1edf-4fbe-9802-7d83169bf764-istiod-ca-cert\") pod \"dcca471e-1edf-4fbe-9802-7d83169bf764\" (UID: \"dcca471e-1edf-4fbe-9802-7d83169bf764\") "
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463810    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a71d461-c056-47b8-ae80-a12994784a4e-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463830    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463841    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463852    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a71d461-c056-47b8-ae80-a12994784a4e-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463863    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463874    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef3ee10-d844-498e-be25-9f151d022caa-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463887    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a71d461-c056-47b8-ae80-a12994784a4e-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463898    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a71d461-c056-47b8-ae80-a12994784a4e-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463909    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgd5d\" (UniqueName: \"kubernetes.io/projected/bef3ee10-d844-498e-be25-9f151d022caa-kube-api-access-wgd5d\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463922    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.463933    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef3ee10-d844-498e-be25-9f151d022caa-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.464304    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcca471e-1edf-4fbe-9802-7d83169bf764-istiod-ca-cert" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:42.465403 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.464550    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-data" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.469843 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.469560    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-certs" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.477233 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.475174    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-credential-socket" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.483851 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.483439    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-socket" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.484256 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.484218    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-token" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:42.485926 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.485885    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-kube-api-access-6chzg" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "kube-api-access-6chzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:42.490137 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.488152    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-envoy" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:42.490137 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.490018    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-podinfo" pod "dcca471e-1edf-4fbe-9802-7d83169bf764" (UID: "dcca471e-1edf-4fbe-9802-7d83169bf764"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:42.490335 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.490312    4742 generic.go:356] "Generic (PLEG): container finished" podID="bef3ee10-d844-498e-be25-9f151d022caa" containerID="87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe" exitCode=0
Oct 29 22:59:42.502113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.490412    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" event={"ID":"bef3ee10-d844-498e-be25-9f151d022caa","Type":"ContainerDied","Data":"87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe"}
Oct 29 22:59:42.502113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.490472    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj" event={"ID":"bef3ee10-d844-498e-be25-9f151d022caa","Type":"ContainerDied","Data":"799aa28db88d35531419ce9527f2a3833792a045d4602bfac9e1b97e45902656"}
Oct 29 22:59:42.502113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.490507    4742 scope.go:122] "RemoveContainer" containerID="87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe"
Oct 29 22:59:42.502113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.490511    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"
Oct 29 22:59:42.502113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.493049    4742 generic.go:356] "Generic (PLEG): container finished" podID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerID="199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116" exitCode=0
Oct 29 22:59:42.502113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.493295    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" event={"ID":"dcca471e-1edf-4fbe-9802-7d83169bf764","Type":"ContainerDied","Data":"199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116"}
Oct 29 22:59:42.502113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.493348    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk" event={"ID":"dcca471e-1edf-4fbe-9802-7d83169bf764","Type":"ContainerDied","Data":"4593ceeeb8bca889dc39aa1662cd9f09bee9825d629c5314f10b95908c149d08"}
Oct 29 22:59:42.502113 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.493434    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"
Oct 29 22:59:42.511130 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.507605    4742 generic.go:356] "Generic (PLEG): container finished" podID="6a71d461-c056-47b8-ae80-a12994784a4e" containerID="338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa" exitCode=0
Oct 29 22:59:42.511130 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.507824    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" event={"ID":"6a71d461-c056-47b8-ae80-a12994784a4e","Type":"ContainerDied","Data":"338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa"}
Oct 29 22:59:42.511130 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.507870    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj" event={"ID":"6a71d461-c056-47b8-ae80-a12994784a4e","Type":"ContainerDied","Data":"2f302184b5f74ecb8c992d6091184a06c20c6310147854e96e846623e91293c5"}
Oct 29 22:59:42.511130 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.508069    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"
Oct 29 22:59:42.522077 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.522054    4742 scope.go:122] "RemoveContainer" containerID="87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe"
Oct 29 22:59:42.522462 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:42.522443    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe\": container with ID starting with 87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe not found: ID does not exist" containerID="87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe"
Oct 29 22:59:42.522539 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.522471    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe"} err="failed to get container status \"87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe\": rpc error: code = NotFound desc = could not find container \"87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe\": container with ID starting with 87f7cd3abba2b2d0dd859f62ec331519ecdec8aa1e83a862e900258d3e6ba7fe not found: ID does not exist"
Oct 29 22:59:42.522539 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.522490    4742 scope.go:122] "RemoveContainer" containerID="199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116"
Oct 29 22:59:42.539565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.535985    4742 scope.go:122] "RemoveContainer" containerID="199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116"
Oct 29 22:59:42.539565 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:42.537074    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116\": container with ID starting with 199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116 not found: ID does not exist" containerID="199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116"
Oct 29 22:59:42.539565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.537124    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116"} err="failed to get container status \"199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116\": rpc error: code = NotFound desc = could not find container \"199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116\": container with ID starting with 199c22aeee1a5312f49e165ec7e2c68d0d2a84132a2602a43ac3b6015ce62116 not found: ID does not exist"
Oct 29 22:59:42.539565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.537151    4742 scope.go:122] "RemoveContainer" containerID="338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa"
Oct 29 22:59:42.574984 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.574954    4742 scope.go:122] "RemoveContainer" containerID="338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa"
Oct 29 22:59:42.575458 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:42.575372    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa\": container with ID starting with 338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa not found: ID does not exist" containerID="338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa"
Oct 29 22:59:42.575458 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.575409    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa"} err="failed to get container status \"338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa\": rpc error: code = NotFound desc = could not find container \"338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa\": container with ID starting with 338bf852bdc785dfc57f5a81c8abeed7d27d5457262324c79a15f8818d4bb4fa not found: ID does not exist"
Oct 29 22:59:42.576627 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.576600    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.577735 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.577690    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"]
Oct 29 22:59:42.580168 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.580143    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.580257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.580175    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.580257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.580190    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6chzg\" (UniqueName: \"kubernetes.io/projected/dcca471e-1edf-4fbe-9802-7d83169bf764-kube-api-access-6chzg\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.580257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.580220    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.580257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.580235    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.580257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.580249    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcca471e-1edf-4fbe-9802-7d83169bf764-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.580257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.580262    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.580257 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.580275    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcca471e-1edf-4fbe-9802-7d83169bf764-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:42.593114 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.589270    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mtdrj"]
Oct 29 22:59:42.625529 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.625493    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"]
Oct 29 22:59:42.639222 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.639192    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-gnhjj"]
Oct 29 22:59:42.656712 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.656676    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"]
Oct 29 22:59:42.661865 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.661838    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-mghdk"]
Oct 29 22:59:42.802897 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:59:42.802862    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67aa34c6_25e3_4dc4_a57e_1dd90de4444c.slice/crio-6e12c15634b4ba9df897b6d3ab416fc8cfd80e13d1f6a14815174a11a8da5c26 WatchSource:0}: Error finding container 6e12c15634b4ba9df897b6d3ab416fc8cfd80e13d1f6a14815174a11a8da5c26: Status 404 returned error can't find the container with id 6e12c15634b4ba9df897b6d3ab416fc8cfd80e13d1f6a14815174a11a8da5c26
Oct 29 22:59:42.803381 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.803355    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"]
Oct 29 22:59:42.805480 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.805450    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:59:42.805551 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.805512    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:59:42.805551 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.805543    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:59:42.997323 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.997288    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" path="/var/lib/kubelet/pods/6a71d461-c056-47b8-ae80-a12994784a4e/volumes"
Oct 29 22:59:42.998360 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.998335    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef3ee10-d844-498e-be25-9f151d022caa" path="/var/lib/kubelet/pods/bef3ee10-d844-498e-be25-9f151d022caa/volumes"
Oct 29 22:59:42.999149 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:42.999129    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" path="/var/lib/kubelet/pods/dcca471e-1edf-4fbe-9802-7d83169bf764/volumes"
Oct 29 22:59:43.228255 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:43.228211    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-01e33b3d-5584ffdb68-mshz5" podUID="f165aa12-7e76-424b-b4ec-02b808b8b281" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.100:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:43.445306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:43.445277    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 22:59:43.519757 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:43.519667    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" event={"ID":"67aa34c6-25e3-4dc4-a57e-1dd90de4444c","Type":"ContainerStarted","Data":"4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657"}
Oct 29 22:59:43.519921 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:43.519845    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" event={"ID":"67aa34c6-25e3-4dc4-a57e-1dd90de4444c","Type":"ContainerStarted","Data":"6e12c15634b4ba9df897b6d3ab416fc8cfd80e13d1f6a14815174a11a8da5c26"}
Oct 29 22:59:43.564216 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:43.564157    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" podStartSLOduration=1.564141197 podStartE2EDuration="1.564141197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:43.562711947 +0000 UTC m=+1040.703453529" watchObservedRunningTime="2026-10-29 22:59:43.564141197 +0000 UTC m=+1040.704882777"
Oct 29 22:59:44.037095 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:44.037045    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"]
Oct 29 22:59:44.042467 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:44.042435    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d99cc79-sh62j"]
Oct 29 22:59:44.430924 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:44.430884    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:44.436460 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:44.436434    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:44.530647 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:44.530616    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:44.531558 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:44.531541    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 22:59:44.994230 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:44.994192    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1" path="/var/lib/kubelet/pods/5d7b19ca-cd3d-4ffa-8dca-ad476791b4d1/volumes"
Oct 29 22:59:46.303743 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.303677    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:59:46.437785 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.437746    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/634172d1-c4be-49b8-993d-7fcbc9792698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634172d1-c4be-49b8-993d-7fcbc9792698-istiod-ca-cert\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.437987 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.437820    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-credential-socket\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.437987 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.437877    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-socket\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.437987 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.437926    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-certs\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.437987 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.437979    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/634172d1-c4be-49b8-993d-7fcbc9792698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634172d1-c4be-49b8-993d-7fcbc9792698-istio-podinfo\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.438687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438034    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-kube-api-access-wsgl7\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-kube-api-access-wsgl7\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.438687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438078    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-istio-token\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-istio-token\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.438687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438151    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-envoy\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.438687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438221    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-data\") pod \"634172d1-c4be-49b8-993d-7fcbc9792698\" (UID: \"634172d1-c4be-49b8-993d-7fcbc9792698\") "
Oct 29 22:59:46.438687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438292    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-socket" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:46.438687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438323    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634172d1-c4be-49b8-993d-7fcbc9792698-istiod-ca-cert" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:46.438687 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438640    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-data" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:46.439057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438764    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-certs" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:46.439057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438883    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.439057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438905    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634172d1-c4be-49b8-993d-7fcbc9792698-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.439057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438920    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.439057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438925    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-credential-socket" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:46.439057 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.438933    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.440498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.440470    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-istio-token" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:46.440736 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.440707    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/634172d1-c4be-49b8-993d-7fcbc9792698-istio-podinfo" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:46.440847 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.440787    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-envoy" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:46.441035 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.441018    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-kube-api-access-wsgl7" pod "634172d1-c4be-49b8-993d-7fcbc9792698" (UID: "634172d1-c4be-49b8-993d-7fcbc9792698"). InnerVolumeSpecName "kube-api-access-wsgl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:46.540612 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.540582    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.540612 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.540609    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634172d1-c4be-49b8-993d-7fcbc9792698-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.540820 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.540620    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsgl7\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-kube-api-access-wsgl7\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.540820 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.540630    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634172d1-c4be-49b8-993d-7fcbc9792698-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.540820 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.540638    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634172d1-c4be-49b8-993d-7fcbc9792698-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 22:59:46.547274 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.547244    4742 generic.go:356] "Generic (PLEG): container finished" podID="634172d1-c4be-49b8-993d-7fcbc9792698" containerID="b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2" exitCode=0
Oct 29 22:59:46.547413 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.547282    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" event={"ID":"634172d1-c4be-49b8-993d-7fcbc9792698","Type":"ContainerDied","Data":"b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2"}
Oct 29 22:59:46.547413 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.547359    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79" event={"ID":"634172d1-c4be-49b8-993d-7fcbc9792698","Type":"ContainerDied","Data":"f7e0d41f71d8fa5d302c6904e9b878b972741198cb5f2c81b9c8c6e10a121729"}
Oct 29 22:59:46.547413 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.547294    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"
Oct 29 22:59:46.547413 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.547388    4742 scope.go:122] "RemoveContainer" containerID="b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2"
Oct 29 22:59:46.558937 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.558919    4742 scope.go:122] "RemoveContainer" containerID="6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d"
Oct 29 22:59:46.568336 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.568320    4742 scope.go:122] "RemoveContainer" containerID="b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2"
Oct 29 22:59:46.568598 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:46.568582    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2\": container with ID starting with b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2 not found: ID does not exist" containerID="b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2"
Oct 29 22:59:46.568641 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.568607    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2"} err="failed to get container status \"b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2\": rpc error: code = NotFound desc = could not find container \"b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2\": container with ID starting with b419165dd04852a5f7ae49a913bfd8e63f3faba9da2607a322739d15ed6ce1d2 not found: ID does not exist"
Oct 29 22:59:46.568641 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.568623    4742 scope.go:122] "RemoveContainer" containerID="6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d"
Oct 29 22:59:46.568868 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:46.568848    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d\": container with ID starting with 6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d not found: ID does not exist" containerID="6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d"
Oct 29 22:59:46.568918 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.568871    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d"} err="failed to get container status \"6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d\": rpc error: code = NotFound desc = could not find container \"6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d\": container with ID starting with 6d50b25e92b0a731bd2a1b179369d03b136810373ab9fcfdcc9cab706e91106d not found: ID does not exist"
Oct 29 22:59:46.599539 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.599512    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"]
Oct 29 22:59:46.605049 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.605025    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-01e33b3d-gw-openshift-default-5c5b5bbb9c-gsh79"]
Oct 29 22:59:46.992222 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:46.992191    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" path="/var/lib/kubelet/pods/634172d1-c4be-49b8-993d-7fcbc9792698/volumes"
Oct 29 22:59:47.160672 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.160640    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"]
Oct 29 22:59:47.161233 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161218    4742 cpu_manager.go:414] "containerMap: removing container" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy"
Oct 29 22:59:47.161306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161234    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy"
Oct 29 22:59:47.161306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161253    4742 cpu_manager.go:414] "containerMap: removing container" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy"
Oct 29 22:59:47.161306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161258    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy"
Oct 29 22:59:47.161306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161278    4742 cpu_manager.go:414] "containerMap: removing container" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy"
Oct 29 22:59:47.161306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161282    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy"
Oct 29 22:59:47.161306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161298    4742 cpu_manager.go:414] "containerMap: removing container" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy"
Oct 29 22:59:47.161306 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161302    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy"
Oct 29 22:59:47.161498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161386    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcca471e-1edf-4fbe-9802-7d83169bf764" containerName="istio-proxy"
Oct 29 22:59:47.161498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161393    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy"
Oct 29 22:59:47.161498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161402    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef3ee10-d844-498e-be25-9f151d022caa" containerName="istio-proxy"
Oct 29 22:59:47.161498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161411    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a71d461-c056-47b8-ae80-a12994784a4e" containerName="istio-proxy"
Oct 29 22:59:47.161498 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.161418    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy"
Oct 29 22:59:47.166495 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.166479    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:47.188641 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.188617    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-28ed1cea\""
Oct 29 22:59:47.190347 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.190328    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"]
Oct 29 22:59:47.202722 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.202697    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"]
Oct 29 22:59:47.203372 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.203354    4742 cpu_manager.go:414] "containerMap: removing container" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy"
Oct 29 22:59:47.203476 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.203375    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634172d1-c4be-49b8-993d-7fcbc9792698" containerName="istio-proxy"
Oct 29 22:59:47.206851 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.206833    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:47.223221 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.223200    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-28ed1cea-dockercfg-z9s5m\""
Oct 29 22:59:47.237311 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.237281    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"]
Oct 29 22:59:47.248990 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.248927    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxk5\" (UniqueName: \"kubernetes.io/projected/bc8cfc30-4859-4b97-8375-aa2ec37a009f-kube-api-access-2hxk5\") pod \"payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh\" (UID: \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:47.249132 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.249040    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95dd232-389e-456d-accb-65be45758b49-plugins-config-volume\") pod \"payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9\" (UID: \"a95dd232-389e-456d-accb-65be45758b49\") " pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:47.249248 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.249225    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrz6\" (UniqueName: \"kubernetes.io/projected/a95dd232-389e-456d-accb-65be45758b49-kube-api-access-bwrz6\") pod \"payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9\" (UID: \"a95dd232-389e-456d-accb-65be45758b49\") " pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:47.249314 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.249266    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8cfc30-4859-4b97-8375-aa2ec37a009f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh\" (UID: \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:47.354434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.351064    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxk5\" (UniqueName: \"kubernetes.io/projected/bc8cfc30-4859-4b97-8375-aa2ec37a009f-kube-api-access-2hxk5\") pod \"payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh\" (UID: \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:47.354434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.351205    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95dd232-389e-456d-accb-65be45758b49-plugins-config-volume\") pod \"payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9\" (UID: \"a95dd232-389e-456d-accb-65be45758b49\") " pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:47.354434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.351345    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrz6\" (UniqueName: \"kubernetes.io/projected/a95dd232-389e-456d-accb-65be45758b49-kube-api-access-bwrz6\") pod \"payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9\" (UID: \"a95dd232-389e-456d-accb-65be45758b49\") " pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:47.354434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.351373    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8cfc30-4859-4b97-8375-aa2ec37a009f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh\" (UID: \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:47.354434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.352360    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8cfc30-4859-4b97-8375-aa2ec37a009f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh\" (UID: \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:47.354434 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.353729    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95dd232-389e-456d-accb-65be45758b49-plugins-config-volume\") pod \"payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9\" (UID: \"a95dd232-389e-456d-accb-65be45758b49\") " pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:47.407107 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.405933    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxk5\" (UniqueName: \"kubernetes.io/projected/bc8cfc30-4859-4b97-8375-aa2ec37a009f-kube-api-access-2hxk5\") pod \"payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh\" (UID: \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:47.430145 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.421298    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrz6\" (UniqueName: \"kubernetes.io/projected/a95dd232-389e-456d-accb-65be45758b49-kube-api-access-bwrz6\") pod \"payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9\" (UID: \"a95dd232-389e-456d-accb-65be45758b49\") " pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:47.479983 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.479952    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:47.520790 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.520761    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:47.794821 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:59:47.794786    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8cfc30_4859_4b97_8375_aa2ec37a009f.slice/crio-f5eda2ae8d25269835931e2b6fbac88055a7d7272f0365306b61fd2cbda99e94 WatchSource:0}: Error finding container f5eda2ae8d25269835931e2b6fbac88055a7d7272f0365306b61fd2cbda99e94: Status 404 returned error can't find the container with id f5eda2ae8d25269835931e2b6fbac88055a7d7272f0365306b61fd2cbda99e94
Oct 29 22:59:47.796122 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.796097    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"]
Oct 29 22:59:47.841755 ip-10-0-139-130 kubenswrapper[4742]: W1029 22:59:47.841728    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95dd232_389e_456d_accb_65be45758b49.slice/crio-43b9e15fe11465d2a3a6de70b669f2a0132f4ca046bdd4cda072c480dd4050b7 WatchSource:0}: Error finding container 43b9e15fe11465d2a3a6de70b669f2a0132f4ca046bdd4cda072c480dd4050b7: Status 404 returned error can't find the container with id 43b9e15fe11465d2a3a6de70b669f2a0132f4ca046bdd4cda072c480dd4050b7
Oct 29 22:59:47.842896 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:47.842876    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"]
Oct 29 22:59:48.584601 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:48.584562    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9" event={"ID":"a95dd232-389e-456d-accb-65be45758b49","Type":"ContainerStarted","Data":"59e910364a985b1e3bacf651a79f6df74afe178917ab6cdea8c30630ed01d68c"}
Oct 29 22:59:48.585156 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:48.584662    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9" event={"ID":"a95dd232-389e-456d-accb-65be45758b49","Type":"ContainerStarted","Data":"43b9e15fe11465d2a3a6de70b669f2a0132f4ca046bdd4cda072c480dd4050b7"}
Oct 29 22:59:48.585156 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:48.584809    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 22:59:48.586453 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:48.586429    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh" event={"ID":"bc8cfc30-4859-4b97-8375-aa2ec37a009f","Type":"ContainerStarted","Data":"24facfedb2b5a9277ca427a9a8a08c0e2df0ea3f40553191ac5f7dc772cf0f6d"}
Oct 29 22:59:48.586565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:48.586475    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh" event={"ID":"bc8cfc30-4859-4b97-8375-aa2ec37a009f","Type":"ContainerStarted","Data":"f5eda2ae8d25269835931e2b6fbac88055a7d7272f0365306b61fd2cbda99e94"}
Oct 29 22:59:48.586565 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:48.586515    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:48.630875 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:48.630824    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9" podStartSLOduration=1.630809829 podStartE2EDuration="1.630809829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:48.625316405 +0000 UTC m=+1045.766057990" watchObservedRunningTime="2026-10-29 22:59:48.630809829 +0000 UTC m=+1045.771551412"
Oct 29 22:59:48.665896 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:48.665834    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh" podStartSLOduration=1.665815201 podStartE2EDuration="1.665815201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:48.663478973 +0000 UTC m=+1045.804220558" watchObservedRunningTime="2026-10-29 22:59:48.665815201 +0000 UTC m=+1045.806556786"
Oct 29 22:59:50.028582 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:50.028543    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"]
Oct 29 22:59:50.040857 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:50.040829    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf3b816-2kdxx"]
Oct 29 22:59:50.999271 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:50.999235    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b443449d-5949-4f91-bb46-9e786c419e0b" path="/var/lib/kubelet/pods/b443449d-5949-4f91-bb46-9e786c419e0b/volumes"
Oct 29 22:59:53.128495 ip-10-0-139-130 kubenswrapper[4742]: E1029 22:59:53.128459    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 22:59:56.034899 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:56.034857    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"]
Oct 29 22:59:56.044207 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:56.044167    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592c9d23-lqfxq"]
Oct 29 22:59:56.992097 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:56.992062    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62950ab1-2f2c-4966-99d5-08cc77b16da8" path="/var/lib/kubelet/pods/62950ab1-2f2c-4966-99d5-08cc77b16da8/volumes"
Oct 29 22:59:59.597195 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:59.597168    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 22:59:59.597583 ip-10-0-139-130 kubenswrapper[4742]: I1029 22:59:59.597232    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 23:00:00.157865 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.157818    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"]
Oct 29 23:00:00.163435 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.163408    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:00.181870 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.181837    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"]
Oct 29 23:00:00.186902 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.186881    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"]
Oct 29 23:00:00.187002 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.186984    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:00.191052 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.191024    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:00.201179 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.201150    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"]
Oct 29 23:00:00.205412 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.205385    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:00:00.207705 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.207680    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"]
Oct 29 23:00:00.226016 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.225985    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"]
Oct 29 23:00:00.227258 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.227236    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:00.236370 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.236346    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:00.242541 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.236253    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"]
Oct 29 23:00:00.242770 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.242755    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"]
Oct 29 23:00:00.250740 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.250709    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5gq\" (UniqueName: \"kubernetes.io/projected/be0273fd-b084-47ee-86e0-89fcb415793a-kube-api-access-rb5gq\") pod \"maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd\" (UID: \"be0273fd-b084-47ee-86e0-89fcb415793a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:00.250934 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.250909    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0273fd-b084-47ee-86e0-89fcb415793a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd\" (UID: \"be0273fd-b084-47ee-86e0-89fcb415793a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:00.251011 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.250987    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de665f9-a7b0-46db-8e52-eacd8612aa4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh\" (UID: \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:00.251162 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.251145    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/8de665f9-a7b0-46db-8e52-eacd8612aa4d-kube-api-access-m5m75\") pod \"maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh\" (UID: \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:00.251261 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.251245    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/930d7aea-d138-4786-ad20-850bc077cb9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-hfcjp\" (UID: \"930d7aea-d138-4786-ad20-850bc077cb9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:00.251348 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.251336    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zww\" (UniqueName: \"kubernetes.io/projected/930d7aea-d138-4786-ad20-850bc077cb9a-kube-api-access-j7zww\") pod \"maas-api-key-cleanup-29888580-hfcjp\" (UID: \"930d7aea-d138-4786-ad20-850bc077cb9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:00.251472 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.251456    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"]
Oct 29 23:00:00.257581 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.257552    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"]
Oct 29 23:00:00.352352 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.352315    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5gq\" (UniqueName: \"kubernetes.io/projected/be0273fd-b084-47ee-86e0-89fcb415793a-kube-api-access-rb5gq\") pod \"maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd\" (UID: \"be0273fd-b084-47ee-86e0-89fcb415793a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:00.352571 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.352383    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r88d2\" (UniqueName: \"kubernetes.io/projected/e8f181f5-a403-425a-83ff-f9eb52eed205-kube-api-access-r88d2\") pod \"maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn\" (UID: \"e8f181f5-a403-425a-83ff-f9eb52eed205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:00.352571 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.352517    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0273fd-b084-47ee-86e0-89fcb415793a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd\" (UID: \"be0273fd-b084-47ee-86e0-89fcb415793a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:00.352735 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.352599    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb03fa-2ed7-4739-9536-901f5fece9c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p\" (UID: \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:00.352735 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.352633    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de665f9-a7b0-46db-8e52-eacd8612aa4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh\" (UID: \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:00.352873 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.352846    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfk27\" (UniqueName: \"kubernetes.io/projected/e4cb03fa-2ed7-4739-9536-901f5fece9c4-kube-api-access-dfk27\") pod \"maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p\" (UID: \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:00.353024 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.352897    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/8de665f9-a7b0-46db-8e52-eacd8612aa4d-kube-api-access-m5m75\") pod \"maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh\" (UID: \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:00.353024 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.353015    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f181f5-a403-425a-83ff-f9eb52eed205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn\" (UID: \"e8f181f5-a403-425a-83ff-f9eb52eed205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:00.353205 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.353043    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/930d7aea-d138-4786-ad20-850bc077cb9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-hfcjp\" (UID: \"930d7aea-d138-4786-ad20-850bc077cb9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:00.353205 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.353191    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zww\" (UniqueName: \"kubernetes.io/projected/930d7aea-d138-4786-ad20-850bc077cb9a-kube-api-access-j7zww\") pod \"maas-api-key-cleanup-29888580-hfcjp\" (UID: \"930d7aea-d138-4786-ad20-850bc077cb9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:00.353535 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.353512    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de665f9-a7b0-46db-8e52-eacd8612aa4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh\" (UID: \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:00.353687 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.353564    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0273fd-b084-47ee-86e0-89fcb415793a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd\" (UID: \"be0273fd-b084-47ee-86e0-89fcb415793a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:00.354171 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.354147    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/930d7aea-d138-4786-ad20-850bc077cb9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-hfcjp\" (UID: \"930d7aea-d138-4786-ad20-850bc077cb9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:00.380775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.380746    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/8de665f9-a7b0-46db-8e52-eacd8612aa4d-kube-api-access-m5m75\") pod \"maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh\" (UID: \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:00.380995 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.380892    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5gq\" (UniqueName: \"kubernetes.io/projected/be0273fd-b084-47ee-86e0-89fcb415793a-kube-api-access-rb5gq\") pod \"maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd\" (UID: \"be0273fd-b084-47ee-86e0-89fcb415793a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:00.381211 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.381190    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zww\" (UniqueName: \"kubernetes.io/projected/930d7aea-d138-4786-ad20-850bc077cb9a-kube-api-access-j7zww\") pod \"maas-api-key-cleanup-29888580-hfcjp\" (UID: \"930d7aea-d138-4786-ad20-850bc077cb9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:00.454935 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.454837    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfk27\" (UniqueName: \"kubernetes.io/projected/e4cb03fa-2ed7-4739-9536-901f5fece9c4-kube-api-access-dfk27\") pod \"maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p\" (UID: \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:00.454935 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.454928    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f181f5-a403-425a-83ff-f9eb52eed205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn\" (UID: \"e8f181f5-a403-425a-83ff-f9eb52eed205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:00.455231 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.455007    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r88d2\" (UniqueName: \"kubernetes.io/projected/e8f181f5-a403-425a-83ff-f9eb52eed205-kube-api-access-r88d2\") pod \"maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn\" (UID: \"e8f181f5-a403-425a-83ff-f9eb52eed205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:00.455231 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.455044    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb03fa-2ed7-4739-9536-901f5fece9c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p\" (UID: \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:00.455610 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.455588    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f181f5-a403-425a-83ff-f9eb52eed205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn\" (UID: \"e8f181f5-a403-425a-83ff-f9eb52eed205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:00.455699 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.455681    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb03fa-2ed7-4739-9536-901f5fece9c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p\" (UID: \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:00.477152 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.477072    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r88d2\" (UniqueName: \"kubernetes.io/projected/e8f181f5-a403-425a-83ff-f9eb52eed205-kube-api-access-r88d2\") pod \"maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn\" (UID: \"e8f181f5-a403-425a-83ff-f9eb52eed205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:00.477305 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.477180    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfk27\" (UniqueName: \"kubernetes.io/projected/e4cb03fa-2ed7-4739-9536-901f5fece9c4-kube-api-access-dfk27\") pod \"maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p\" (UID: \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:00.478883 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.478862    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:00.522314 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.522284    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:00.530609 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.530580    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:00.553955 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.553919    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:00.569497 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.569456    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:00.793778 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.793750    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"]
Oct 29 23:00:00.855377 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.855351    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"]
Oct 29 23:00:00.855482 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:00.855349    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de665f9_a7b0_46db_8e52_eacd8612aa4d.slice/crio-302b6c65bb2caf8ace8dd8735ac06bb0d8b7d683d14c0d40f1a6b7b33773c87d WatchSource:0}: Error finding container 302b6c65bb2caf8ace8dd8735ac06bb0d8b7d683d14c0d40f1a6b7b33773c87d: Status 404 returned error can't find the container with id 302b6c65bb2caf8ace8dd8735ac06bb0d8b7d683d14c0d40f1a6b7b33773c87d
Oct 29 23:00:00.869131 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.867651    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"]
Oct 29 23:00:00.877356 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:00.877321    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0273fd_b084_47ee_86e0_89fcb415793a.slice/crio-a7d342f68f51e56ee8632680e14ce799c3801749f99ed11c651f63d5546a6582 WatchSource:0}: Error finding container a7d342f68f51e56ee8632680e14ce799c3801749f99ed11c651f63d5546a6582: Status 404 returned error can't find the container with id a7d342f68f51e56ee8632680e14ce799c3801749f99ed11c651f63d5546a6582
Oct 29 23:00:00.907576 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:00.907540    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cb03fa_2ed7_4739_9536_901f5fece9c4.slice/crio-e4bd634d0041cdb7146ff74c7b3dae7a59b594791cf58679513a322be73eea34 WatchSource:0}: Error finding container e4bd634d0041cdb7146ff74c7b3dae7a59b594791cf58679513a322be73eea34: Status 404 returned error can't find the container with id e4bd634d0041cdb7146ff74c7b3dae7a59b594791cf58679513a322be73eea34
Oct 29 23:00:00.908371 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.908347    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"]
Oct 29 23:00:00.935319 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:00.935293    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"]
Oct 29 23:00:00.943461 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:00.943434    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f181f5_a403_425a_83ff_f9eb52eed205.slice/crio-c2bd7b4af6cbaee57a6e22fd058a81e73d1e7f8135e5743b12264bcc4d629bab WatchSource:0}: Error finding container c2bd7b4af6cbaee57a6e22fd058a81e73d1e7f8135e5743b12264bcc4d629bab: Status 404 returned error can't find the container with id c2bd7b4af6cbaee57a6e22fd058a81e73d1e7f8135e5743b12264bcc4d629bab
Oct 29 23:00:01.694157 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.694047    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" event={"ID":"be0273fd-b084-47ee-86e0-89fcb415793a","Type":"ContainerStarted","Data":"1937ac755d78f378058bb1a721e30a27eabca36315bd2b25b88d72d29988be30"}
Oct 29 23:00:01.694324 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.694159    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" event={"ID":"be0273fd-b084-47ee-86e0-89fcb415793a","Type":"ContainerStarted","Data":"a7d342f68f51e56ee8632680e14ce799c3801749f99ed11c651f63d5546a6582"}
Oct 29 23:00:01.695873 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.695841    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" event={"ID":"e4cb03fa-2ed7-4739-9536-901f5fece9c4","Type":"ContainerStarted","Data":"21adb0d13ddb293cb364f81bfbe338779849cb8c26720dae178e39a8423bf0df"}
Oct 29 23:00:01.696016 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.695890    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" event={"ID":"e4cb03fa-2ed7-4739-9536-901f5fece9c4","Type":"ContainerStarted","Data":"e4bd634d0041cdb7146ff74c7b3dae7a59b594791cf58679513a322be73eea34"}
Oct 29 23:00:01.697630 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.697605    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" event={"ID":"e8f181f5-a403-425a-83ff-f9eb52eed205","Type":"ContainerStarted","Data":"d402b1bf3075f19338709707bbdb46a4d98c73d062dc8542fb839300efafa787"}
Oct 29 23:00:01.697783 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.697759    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" event={"ID":"e8f181f5-a403-425a-83ff-f9eb52eed205","Type":"ContainerStarted","Data":"c2bd7b4af6cbaee57a6e22fd058a81e73d1e7f8135e5743b12264bcc4d629bab"}
Oct 29 23:00:01.699464 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.699439    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" event={"ID":"8de665f9-a7b0-46db-8e52-eacd8612aa4d","Type":"ContainerStarted","Data":"43d9d33287f1a05e2036835d62b31e5582b235d40a9764dce538931b6d5fb113"}
Oct 29 23:00:01.699574 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.699485    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" event={"ID":"8de665f9-a7b0-46db-8e52-eacd8612aa4d","Type":"ContainerStarted","Data":"302b6c65bb2caf8ace8dd8735ac06bb0d8b7d683d14c0d40f1a6b7b33773c87d"}
Oct 29 23:00:01.701121 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.701071    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" event={"ID":"930d7aea-d138-4786-ad20-850bc077cb9a","Type":"ContainerStarted","Data":"0544a66396cd6e6e58d7eb4a27bff3c2431938d7870ffa0d75a2990e79b7a0da"}
Oct 29 23:00:01.701190 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.701141    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" event={"ID":"930d7aea-d138-4786-ad20-850bc077cb9a","Type":"ContainerStarted","Data":"5c5f9aa7c5e9f97acda0eeeec7ed156b41bc7781ca97b8f9dc0e466e7df49e54"}
Oct 29 23:00:01.740250 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.740190    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" podStartSLOduration=1.740176527 podStartE2EDuration="1.740176527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.735821305 +0000 UTC m=+1058.876562890" watchObservedRunningTime="2026-10-29 23:00:01.740176527 +0000 UTC m=+1058.880918111"
Oct 29 23:00:01.776938 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.776883    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" podStartSLOduration=1.776869322 podStartE2EDuration="1.776869322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.774802503 +0000 UTC m=+1058.915544087" watchObservedRunningTime="2026-10-29 23:00:01.776869322 +0000 UTC m=+1058.917610939"
Oct 29 23:00:01.815072 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.815020    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" podStartSLOduration=1.815006263 podStartE2EDuration="1.815006263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.811736546 +0000 UTC m=+1058.952478132" watchObservedRunningTime="2026-10-29 23:00:01.815006263 +0000 UTC m=+1058.955747846"
Oct 29 23:00:01.855498 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.855443    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" podStartSLOduration=1.855428529 podStartE2EDuration="1.855428529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.850634628 +0000 UTC m=+1058.991376206" watchObservedRunningTime="2026-10-29 23:00:01.855428529 +0000 UTC m=+1058.996170113"
Oct 29 23:00:01.890406 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:01.890246    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" podStartSLOduration=1.89022735 podStartE2EDuration="1.89022735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.888378854 +0000 UTC m=+1059.029120438" watchObservedRunningTime="2026-10-29 23:00:01.89022735 +0000 UTC m=+1059.030968935"
Oct 29 23:00:02.714541 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.714506    4742 generic.go:356] "Generic (PLEG): container finished" podID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerID="d402b1bf3075f19338709707bbdb46a4d98c73d062dc8542fb839300efafa787" exitCode=60
Oct 29 23:00:02.714742 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.714592    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" event={"ID":"e8f181f5-a403-425a-83ff-f9eb52eed205","Type":"ContainerDied","Data":"d402b1bf3075f19338709707bbdb46a4d98c73d062dc8542fb839300efafa787"}
Oct 29 23:00:02.714921 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.714907    4742 scope.go:122] "RemoveContainer" containerID="d402b1bf3075f19338709707bbdb46a4d98c73d062dc8542fb839300efafa787"
Oct 29 23:00:02.716399 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.716377    4742 generic.go:356] "Generic (PLEG): container finished" podID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerID="43d9d33287f1a05e2036835d62b31e5582b235d40a9764dce538931b6d5fb113" exitCode=60
Oct 29 23:00:02.716488 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.716459    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" event={"ID":"8de665f9-a7b0-46db-8e52-eacd8612aa4d","Type":"ContainerDied","Data":"43d9d33287f1a05e2036835d62b31e5582b235d40a9764dce538931b6d5fb113"}
Oct 29 23:00:02.716705 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.716691    4742 scope.go:122] "RemoveContainer" containerID="43d9d33287f1a05e2036835d62b31e5582b235d40a9764dce538931b6d5fb113"
Oct 29 23:00:02.718361 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.718339    4742 generic.go:356] "Generic (PLEG): container finished" podID="930d7aea-d138-4786-ad20-850bc077cb9a" containerID="0544a66396cd6e6e58d7eb4a27bff3c2431938d7870ffa0d75a2990e79b7a0da" exitCode=60
Oct 29 23:00:02.718594 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.718461    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" event={"ID":"930d7aea-d138-4786-ad20-850bc077cb9a","Type":"ContainerDied","Data":"0544a66396cd6e6e58d7eb4a27bff3c2431938d7870ffa0d75a2990e79b7a0da"}
Oct 29 23:00:02.718650 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.718618    4742 scope.go:122] "RemoveContainer" containerID="0544a66396cd6e6e58d7eb4a27bff3c2431938d7870ffa0d75a2990e79b7a0da"
Oct 29 23:00:02.720162 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.720137    4742 generic.go:356] "Generic (PLEG): container finished" podID="be0273fd-b084-47ee-86e0-89fcb415793a" containerID="1937ac755d78f378058bb1a721e30a27eabca36315bd2b25b88d72d29988be30" exitCode=60
Oct 29 23:00:02.720248 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.720211    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" event={"ID":"be0273fd-b084-47ee-86e0-89fcb415793a","Type":"ContainerDied","Data":"1937ac755d78f378058bb1a721e30a27eabca36315bd2b25b88d72d29988be30"}
Oct 29 23:00:02.720409 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.720395    4742 scope.go:122] "RemoveContainer" containerID="1937ac755d78f378058bb1a721e30a27eabca36315bd2b25b88d72d29988be30"
Oct 29 23:00:02.721760 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.721742    4742 generic.go:356] "Generic (PLEG): container finished" podID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerID="21adb0d13ddb293cb364f81bfbe338779849cb8c26720dae178e39a8423bf0df" exitCode=60
Oct 29 23:00:02.721760 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.721767    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" event={"ID":"e4cb03fa-2ed7-4739-9536-901f5fece9c4","Type":"ContainerDied","Data":"21adb0d13ddb293cb364f81bfbe338779849cb8c26720dae178e39a8423bf0df"}
Oct 29 23:00:02.722052 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:02.722036    4742 scope.go:122] "RemoveContainer" containerID="21adb0d13ddb293cb364f81bfbe338779849cb8c26720dae178e39a8423bf0df"
Oct 29 23:00:03.735824 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:03.735786    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" event={"ID":"e4cb03fa-2ed7-4739-9536-901f5fece9c4","Type":"ContainerStarted","Data":"7550e06b6ece0ff6470416f87e5b17748cf162ba518503eaef74f81d2545dacf"}
Oct 29 23:00:03.737883 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:03.737857    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" event={"ID":"e8f181f5-a403-425a-83ff-f9eb52eed205","Type":"ContainerStarted","Data":"03f20e2b5309c0f4ac22dacfcc320fe6cde9ba1712746e62d87c3711c232c454"}
Oct 29 23:00:03.744662 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:03.743157    4742 generic.go:356] "Generic (PLEG): container finished" podID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerID="9f25f6b3c214cafa39483cded1b7b46ef2bfb27386dd667d21b31738ce1aad6e" exitCode=60
Oct 29 23:00:03.744662 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:03.743273    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" event={"ID":"8de665f9-a7b0-46db-8e52-eacd8612aa4d","Type":"ContainerDied","Data":"9f25f6b3c214cafa39483cded1b7b46ef2bfb27386dd667d21b31738ce1aad6e"}
Oct 29 23:00:03.744662 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:03.743365    4742 scope.go:122] "RemoveContainer" containerID="43d9d33287f1a05e2036835d62b31e5582b235d40a9764dce538931b6d5fb113"
Oct 29 23:00:03.746153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:03.746128    4742 scope.go:122] "RemoveContainer" containerID="9f25f6b3c214cafa39483cded1b7b46ef2bfb27386dd667d21b31738ce1aad6e"
Oct 29 23:00:03.746757 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:03.746730    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh_odh-ai-gateway-infra(8de665f9-a7b0-46db-8e52-eacd8612aa4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d"
Oct 29 23:00:03.749415 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:03.749373    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" event={"ID":"930d7aea-d138-4786-ad20-850bc077cb9a","Type":"ContainerStarted","Data":"e3febcf867674dc88507a22e82a1b872d127a77bf8a7ec1d6b11c1c9cf400474"}
Oct 29 23:00:03.751497 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:03.751473    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" event={"ID":"be0273fd-b084-47ee-86e0-89fcb415793a","Type":"ContainerStarted","Data":"e480dabb79f81a7cabf804331dfe9424f7f9be555c202ef94b01a9f1490f6041"}
Oct 29 23:00:04.761040 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.760999    4742 generic.go:356] "Generic (PLEG): container finished" podID="be0273fd-b084-47ee-86e0-89fcb415793a" containerID="e480dabb79f81a7cabf804331dfe9424f7f9be555c202ef94b01a9f1490f6041" exitCode=60
Oct 29 23:00:04.761560 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.761111    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" event={"ID":"be0273fd-b084-47ee-86e0-89fcb415793a","Type":"ContainerDied","Data":"e480dabb79f81a7cabf804331dfe9424f7f9be555c202ef94b01a9f1490f6041"}
Oct 29 23:00:04.761560 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.761174    4742 scope.go:122] "RemoveContainer" containerID="1937ac755d78f378058bb1a721e30a27eabca36315bd2b25b88d72d29988be30"
Oct 29 23:00:04.761560 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.761496    4742 scope.go:122] "RemoveContainer" containerID="e480dabb79f81a7cabf804331dfe9424f7f9be555c202ef94b01a9f1490f6041"
Oct 29 23:00:04.761908 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:04.761883    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd_odh-ai-gateway-infra(be0273fd-b084-47ee-86e0-89fcb415793a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" podUID="be0273fd-b084-47ee-86e0-89fcb415793a"
Oct 29 23:00:04.763211 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.763187    4742 generic.go:356] "Generic (PLEG): container finished" podID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerID="7550e06b6ece0ff6470416f87e5b17748cf162ba518503eaef74f81d2545dacf" exitCode=60
Oct 29 23:00:04.763308 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.763262    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" event={"ID":"e4cb03fa-2ed7-4739-9536-901f5fece9c4","Type":"ContainerDied","Data":"7550e06b6ece0ff6470416f87e5b17748cf162ba518503eaef74f81d2545dacf"}
Oct 29 23:00:04.763519 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.763502    4742 scope.go:122] "RemoveContainer" containerID="7550e06b6ece0ff6470416f87e5b17748cf162ba518503eaef74f81d2545dacf"
Oct 29 23:00:04.763935 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:04.763913    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p_odh-ai-gateway-infra(e4cb03fa-2ed7-4739-9536-901f5fece9c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4"
Oct 29 23:00:04.765338 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.765314    4742 generic.go:356] "Generic (PLEG): container finished" podID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerID="03f20e2b5309c0f4ac22dacfcc320fe6cde9ba1712746e62d87c3711c232c454" exitCode=60
Oct 29 23:00:04.765459 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.765398    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" event={"ID":"e8f181f5-a403-425a-83ff-f9eb52eed205","Type":"ContainerDied","Data":"03f20e2b5309c0f4ac22dacfcc320fe6cde9ba1712746e62d87c3711c232c454"}
Oct 29 23:00:04.765644 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.765630    4742 scope.go:122] "RemoveContainer" containerID="03f20e2b5309c0f4ac22dacfcc320fe6cde9ba1712746e62d87c3711c232c454"
Oct 29 23:00:04.765921 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:04.765903    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn_odh-ai-gateway-infra(e8f181f5-a403-425a-83ff-f9eb52eed205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205"
Oct 29 23:00:04.767537 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.767521    4742 scope.go:122] "RemoveContainer" containerID="9f25f6b3c214cafa39483cded1b7b46ef2bfb27386dd667d21b31738ce1aad6e"
Oct 29 23:00:04.767852 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:04.767797    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh_odh-ai-gateway-infra(8de665f9-a7b0-46db-8e52-eacd8612aa4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d"
Oct 29 23:00:04.769875 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.769854    4742 generic.go:356] "Generic (PLEG): container finished" podID="930d7aea-d138-4786-ad20-850bc077cb9a" containerID="e3febcf867674dc88507a22e82a1b872d127a77bf8a7ec1d6b11c1c9cf400474" exitCode=60
Oct 29 23:00:04.769957 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.769895    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" event={"ID":"930d7aea-d138-4786-ad20-850bc077cb9a","Type":"ContainerDied","Data":"e3febcf867674dc88507a22e82a1b872d127a77bf8a7ec1d6b11c1c9cf400474"}
Oct 29 23:00:04.770206 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.770191    4742 scope.go:122] "RemoveContainer" containerID="e3febcf867674dc88507a22e82a1b872d127a77bf8a7ec1d6b11c1c9cf400474"
Oct 29 23:00:04.770500 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:04.770478    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-hfcjp_odh-ai-gateway-infra(930d7aea-d138-4786-ad20-850bc077cb9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" podUID="930d7aea-d138-4786-ad20-850bc077cb9a"
Oct 29 23:00:04.775903 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.775883    4742 scope.go:122] "RemoveContainer" containerID="21adb0d13ddb293cb364f81bfbe338779849cb8c26720dae178e39a8423bf0df"
Oct 29 23:00:04.790068 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.790045    4742 scope.go:122] "RemoveContainer" containerID="d402b1bf3075f19338709707bbdb46a4d98c73d062dc8542fb839300efafa787"
Oct 29 23:00:04.805894 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:04.805862    4742 scope.go:122] "RemoveContainer" containerID="0544a66396cd6e6e58d7eb4a27bff3c2431938d7870ffa0d75a2990e79b7a0da"
Oct 29 23:00:05.781661 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:05.781632    4742 scope.go:122] "RemoveContainer" containerID="e480dabb79f81a7cabf804331dfe9424f7f9be555c202ef94b01a9f1490f6041"
Oct 29 23:00:05.782141 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:05.782063    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd_odh-ai-gateway-infra(be0273fd-b084-47ee-86e0-89fcb415793a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" podUID="be0273fd-b084-47ee-86e0-89fcb415793a"
Oct 29 23:00:05.783366 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:05.783345    4742 scope.go:122] "RemoveContainer" containerID="7550e06b6ece0ff6470416f87e5b17748cf162ba518503eaef74f81d2545dacf"
Oct 29 23:00:05.783678 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:05.783652    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p_odh-ai-gateway-infra(e4cb03fa-2ed7-4739-9536-901f5fece9c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4"
Oct 29 23:00:05.785169 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:05.785151    4742 scope.go:122] "RemoveContainer" containerID="03f20e2b5309c0f4ac22dacfcc320fe6cde9ba1712746e62d87c3711c232c454"
Oct 29 23:00:05.785454 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:05.785428    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn_odh-ai-gateway-infra(e8f181f5-a403-425a-83ff-f9eb52eed205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205"
Oct 29 23:00:05.786995 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:05.786975    4742 scope.go:122] "RemoveContainer" containerID="e3febcf867674dc88507a22e82a1b872d127a77bf8a7ec1d6b11c1c9cf400474"
Oct 29 23:00:05.787310 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:05.787280    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-hfcjp_odh-ai-gateway-infra(930d7aea-d138-4786-ad20-850bc077cb9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" podUID="930d7aea-d138-4786-ad20-850bc077cb9a"
Oct 29 23:00:07.977781 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:07.977745    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"]
Oct 29 23:00:09.040598 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:09.040556    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"]
Oct 29 23:00:09.050198 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:09.050168    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c16610c5-8h9cl"]
Oct 29 23:00:10.992659 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:10.992628    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ea7194-9f8a-45e8-8660-b3cb2ebea00c" path="/var/lib/kubelet/pods/a5ea7194-9f8a-45e8-8660-b3cb2ebea00c/volumes"
Oct 29 23:00:12.979111 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:12.978160    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw" podUID="ff7d5521-a843-4354-bbb0-c3d042da8f4e" containerName="payload-pre-processing" containerID="cri-o://5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2" gracePeriod=25
Oct 29 23:00:13.318781 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.318753    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 23:00:13.388965 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.388877    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff7d5521-a843-4354-bbb0-c3d042da8f4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7d5521-a843-4354-bbb0-c3d042da8f4e-plugins-config-volume\") pod \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\" (UID: \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\") "
Oct 29 23:00:13.390234 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.389243    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7d5521-a843-4354-bbb0-c3d042da8f4e-kube-api-access-gg57d\" (UniqueName: \"kubernetes.io/projected/ff7d5521-a843-4354-bbb0-c3d042da8f4e-kube-api-access-gg57d\") pod \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\" (UID: \"ff7d5521-a843-4354-bbb0-c3d042da8f4e\") "
Oct 29 23:00:13.390506 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.390460    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7d5521-a843-4354-bbb0-c3d042da8f4e-plugins-config-volume" pod "ff7d5521-a843-4354-bbb0-c3d042da8f4e" (UID: "ff7d5521-a843-4354-bbb0-c3d042da8f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.390639 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.390617    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7d5521-a843-4354-bbb0-c3d042da8f4e-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.393295 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.393264    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7d5521-a843-4354-bbb0-c3d042da8f4e-kube-api-access-gg57d" pod "ff7d5521-a843-4354-bbb0-c3d042da8f4e" (UID: "ff7d5521-a843-4354-bbb0-c3d042da8f4e"). InnerVolumeSpecName "kube-api-access-gg57d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.491772 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.491729    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg57d\" (UniqueName: \"kubernetes.io/projected/ff7d5521-a843-4354-bbb0-c3d042da8f4e-kube-api-access-gg57d\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.886934 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.886901    4742 generic.go:356] "Generic (PLEG): container finished" podID="ff7d5521-a843-4354-bbb0-c3d042da8f4e" containerID="5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2" exitCode=0
Oct 29 23:00:13.887152 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.886957    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"
Oct 29 23:00:13.887152 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.886982    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw" event={"ID":"ff7d5521-a843-4354-bbb0-c3d042da8f4e","Type":"ContainerDied","Data":"5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2"}
Oct 29 23:00:13.887152 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.887044    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw" event={"ID":"ff7d5521-a843-4354-bbb0-c3d042da8f4e","Type":"ContainerDied","Data":"7bef9650e736e04ff4aa650f4e116b0ba5f49905c87f5ebf67c87236889ecf83"}
Oct 29 23:00:13.887152 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.887078    4742 scope.go:122] "RemoveContainer" containerID="5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2"
Oct 29 23:00:13.898526 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.898502    4742 scope.go:122] "RemoveContainer" containerID="5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2"
Oct 29 23:00:13.898859 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:13.898836    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2\": container with ID starting with 5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2 not found: ID does not exist" containerID="5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2"
Oct 29 23:00:13.898959 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.898865    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2"} err="failed to get container status \"5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2\": rpc error: code = NotFound desc = could not find container \"5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2\": container with ID starting with 5a2d5e0d48e09b6b4c54251571a721612d4278ab238247625d71314a6aa472e2 not found: ID does not exist"
Oct 29 23:00:13.940942 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.940905    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"]
Oct 29 23:00:13.956066 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:13.956028    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-34d0fd42-57b884ccd7-v5nkw"]
Oct 29 23:00:14.992895 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:14.992864    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7d5521-a843-4354-bbb0-c3d042da8f4e" path="/var/lib/kubelet/pods/ff7d5521-a843-4354-bbb0-c3d042da8f4e/volumes"
Oct 29 23:00:15.984989 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:15.984955    4742 scope.go:122] "RemoveContainer" containerID="e480dabb79f81a7cabf804331dfe9424f7f9be555c202ef94b01a9f1490f6041"
Oct 29 23:00:16.916834 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:16.916777    4742 generic.go:356] "Generic (PLEG): container finished" podID="be0273fd-b084-47ee-86e0-89fcb415793a" containerID="019e56a566e1b28f3936492e998da6634a70aab40ce9bd8ab562e26f83ec270d" exitCode=60
Oct 29 23:00:16.917372 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:16.916857    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" event={"ID":"be0273fd-b084-47ee-86e0-89fcb415793a","Type":"ContainerDied","Data":"019e56a566e1b28f3936492e998da6634a70aab40ce9bd8ab562e26f83ec270d"}
Oct 29 23:00:16.917372 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:16.916933    4742 scope.go:122] "RemoveContainer" containerID="e480dabb79f81a7cabf804331dfe9424f7f9be555c202ef94b01a9f1490f6041"
Oct 29 23:00:16.917481 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:16.917418    4742 scope.go:122] "RemoveContainer" containerID="019e56a566e1b28f3936492e998da6634a70aab40ce9bd8ab562e26f83ec270d"
Oct 29 23:00:16.917823 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:16.917792    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd_odh-ai-gateway-infra(be0273fd-b084-47ee-86e0-89fcb415793a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" podUID="be0273fd-b084-47ee-86e0-89fcb415793a"
Oct 29 23:00:16.994890 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:16.994855    4742 scope.go:122] "RemoveContainer" containerID="9f25f6b3c214cafa39483cded1b7b46ef2bfb27386dd667d21b31738ce1aad6e"
Oct 29 23:00:16.995067 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:16.995051    4742 scope.go:122] "RemoveContainer" containerID="03f20e2b5309c0f4ac22dacfcc320fe6cde9ba1712746e62d87c3711c232c454"
Oct 29 23:00:16.995566 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:16.995541    4742 scope.go:122] "RemoveContainer" containerID="e3febcf867674dc88507a22e82a1b872d127a77bf8a7ec1d6b11c1c9cf400474"
Oct 29 23:00:17.925839 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:17.925802    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" event={"ID":"8de665f9-a7b0-46db-8e52-eacd8612aa4d","Type":"ContainerStarted","Data":"a60deea1332aec1fa490cb344562004b5b6fe9c4884c32563a72f62df33f516d"}
Oct 29 23:00:17.927953 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:17.927931    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" event={"ID":"930d7aea-d138-4786-ad20-850bc077cb9a","Type":"ContainerStarted","Data":"764d23ca2a25ddefc7f944ab48c4bdc7c287a46bf2d72b9edff4e30437a56a5e"}
Oct 29 23:00:17.931548 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:17.931527    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" event={"ID":"e8f181f5-a403-425a-83ff-f9eb52eed205","Type":"ContainerStarted","Data":"ac0146360444484a19bde2d9aa3db879ad5db1d16d882213548ad204d373de52"}
Oct 29 23:00:17.970748 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:17.970715    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"]
Oct 29 23:00:17.989330 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:17.989298    4742 scope.go:122] "RemoveContainer" containerID="7550e06b6ece0ff6470416f87e5b17748cf162ba518503eaef74f81d2545dacf"
Oct 29 23:00:18.269004 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.268971    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:18.373026 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.372965    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0273fd-b084-47ee-86e0-89fcb415793a-kube-api-access-rb5gq\" (UniqueName: \"kubernetes.io/projected/be0273fd-b084-47ee-86e0-89fcb415793a-kube-api-access-rb5gq\") pod \"be0273fd-b084-47ee-86e0-89fcb415793a\" (UID: \"be0273fd-b084-47ee-86e0-89fcb415793a\") "
Oct 29 23:00:18.373259 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.373167    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be0273fd-b084-47ee-86e0-89fcb415793a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0273fd-b084-47ee-86e0-89fcb415793a-maas-api-service-ca\") pod \"be0273fd-b084-47ee-86e0-89fcb415793a\" (UID: \"be0273fd-b084-47ee-86e0-89fcb415793a\") "
Oct 29 23:00:18.373573 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.373540    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0273fd-b084-47ee-86e0-89fcb415793a-maas-api-service-ca" pod "be0273fd-b084-47ee-86e0-89fcb415793a" (UID: "be0273fd-b084-47ee-86e0-89fcb415793a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:18.374010 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.373990    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0273fd-b084-47ee-86e0-89fcb415793a-maas-api-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:18.375296 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.375263    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0273fd-b084-47ee-86e0-89fcb415793a-kube-api-access-rb5gq" pod "be0273fd-b084-47ee-86e0-89fcb415793a" (UID: "be0273fd-b084-47ee-86e0-89fcb415793a"). InnerVolumeSpecName "kube-api-access-rb5gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:18.475027 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.474946    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb5gq\" (UniqueName: \"kubernetes.io/projected/be0273fd-b084-47ee-86e0-89fcb415793a-kube-api-access-rb5gq\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:18.941879 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.941846    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"
Oct 29 23:00:18.941879 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.941871    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd" event={"ID":"be0273fd-b084-47ee-86e0-89fcb415793a","Type":"ContainerDied","Data":"a7d342f68f51e56ee8632680e14ce799c3801749f99ed11c651f63d5546a6582"}
Oct 29 23:00:18.942438 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.941938    4742 scope.go:122] "RemoveContainer" containerID="019e56a566e1b28f3936492e998da6634a70aab40ce9bd8ab562e26f83ec270d"
Oct 29 23:00:18.944330 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.944297    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" event={"ID":"e4cb03fa-2ed7-4739-9536-901f5fece9c4","Type":"ContainerStarted","Data":"86485440481ef79c39c0c1b75a8b627f7b0b91ad0e415e9ec02d050ca90b76a6"}
Oct 29 23:00:18.946657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.946632    4742 generic.go:356] "Generic (PLEG): container finished" podID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerID="ac0146360444484a19bde2d9aa3db879ad5db1d16d882213548ad204d373de52" exitCode=60
Oct 29 23:00:18.946754 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.946701    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" event={"ID":"e8f181f5-a403-425a-83ff-f9eb52eed205","Type":"ContainerDied","Data":"ac0146360444484a19bde2d9aa3db879ad5db1d16d882213548ad204d373de52"}
Oct 29 23:00:18.947045 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.947028    4742 scope.go:122] "RemoveContainer" containerID="ac0146360444484a19bde2d9aa3db879ad5db1d16d882213548ad204d373de52"
Oct 29 23:00:18.947314 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:18.947294    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn_odh-ai-gateway-infra(e8f181f5-a403-425a-83ff-f9eb52eed205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205"
Oct 29 23:00:18.950008 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.949983    4742 generic.go:356] "Generic (PLEG): container finished" podID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerID="a60deea1332aec1fa490cb344562004b5b6fe9c4884c32563a72f62df33f516d" exitCode=60
Oct 29 23:00:18.950124 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.950054    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" event={"ID":"8de665f9-a7b0-46db-8e52-eacd8612aa4d","Type":"ContainerDied","Data":"a60deea1332aec1fa490cb344562004b5b6fe9c4884c32563a72f62df33f516d"}
Oct 29 23:00:18.950409 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.950392    4742 scope.go:122] "RemoveContainer" containerID="a60deea1332aec1fa490cb344562004b5b6fe9c4884c32563a72f62df33f516d"
Oct 29 23:00:18.950750 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:18.950665    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh_odh-ai-gateway-infra(8de665f9-a7b0-46db-8e52-eacd8612aa4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d"
Oct 29 23:00:18.952532 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.952513    4742 generic.go:356] "Generic (PLEG): container finished" podID="930d7aea-d138-4786-ad20-850bc077cb9a" containerID="764d23ca2a25ddefc7f944ab48c4bdc7c287a46bf2d72b9edff4e30437a56a5e" exitCode=60
Oct 29 23:00:18.952622 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.952589    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" event={"ID":"930d7aea-d138-4786-ad20-850bc077cb9a","Type":"ContainerDied","Data":"764d23ca2a25ddefc7f944ab48c4bdc7c287a46bf2d72b9edff4e30437a56a5e"}
Oct 29 23:00:18.952909 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.952894    4742 scope.go:122] "RemoveContainer" containerID="764d23ca2a25ddefc7f944ab48c4bdc7c287a46bf2d72b9edff4e30437a56a5e"
Oct 29 23:00:18.953170 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:18.953154    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-hfcjp_odh-ai-gateway-infra(930d7aea-d138-4786-ad20-850bc077cb9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" podUID="930d7aea-d138-4786-ad20-850bc077cb9a"
Oct 29 23:00:18.954411 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:18.954395    4742 scope.go:122] "RemoveContainer" containerID="03f20e2b5309c0f4ac22dacfcc320fe6cde9ba1712746e62d87c3711c232c454"
Oct 29 23:00:19.006387 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.006356    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"]
Oct 29 23:00:19.016542 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:19.016507    4742 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cb03fa_2ed7_4739_9536_901f5fece9c4.slice/crio-conmon-86485440481ef79c39c0c1b75a8b627f7b0b91ad0e415e9ec02d050ca90b76a6.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:19.033740 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.033578    4742 scope.go:122] "RemoveContainer" containerID="9f25f6b3c214cafa39483cded1b7b46ef2bfb27386dd667d21b31738ce1aad6e"
Oct 29 23:00:19.037301 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.037272    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"]
Oct 29 23:00:19.047798 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.047779    4742 scope.go:122] "RemoveContainer" containerID="e3febcf867674dc88507a22e82a1b872d127a77bf8a7ec1d6b11c1c9cf400474"
Oct 29 23:00:19.087415 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.087384    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"]
Oct 29 23:00:19.273648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.273571    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"]
Oct 29 23:00:19.277581 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.277560    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffba57-29888580-nfxxd"]
Oct 29 23:00:19.962015 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.961924    4742 generic.go:356] "Generic (PLEG): container finished" podID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerID="86485440481ef79c39c0c1b75a8b627f7b0b91ad0e415e9ec02d050ca90b76a6" exitCode=60
Oct 29 23:00:19.962015 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.961992    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" event={"ID":"e4cb03fa-2ed7-4739-9536-901f5fece9c4","Type":"ContainerDied","Data":"86485440481ef79c39c0c1b75a8b627f7b0b91ad0e415e9ec02d050ca90b76a6"}
Oct 29 23:00:19.962616 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.962080    4742 scope.go:122] "RemoveContainer" containerID="7550e06b6ece0ff6470416f87e5b17748cf162ba518503eaef74f81d2545dacf"
Oct 29 23:00:19.962616 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.962567    4742 scope.go:122] "RemoveContainer" containerID="86485440481ef79c39c0c1b75a8b627f7b0b91ad0e415e9ec02d050ca90b76a6"
Oct 29 23:00:19.963043 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:19.962982    4742 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p_odh-ai-gateway-infra(e4cb03fa-2ed7-4739-9536-901f5fece9c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4"
Oct 29 23:00:19.970799 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.970769    4742 generic.go:356] "Generic (PLEG): container finished" podID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerID="4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657" exitCode=0
Oct 29 23:00:19.970921 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.970816    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" event={"ID":"67aa34c6-25e3-4dc4-a57e-1dd90de4444c","Type":"ContainerDied","Data":"4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657"}
Oct 29 23:00:19.971348 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.971328    4742 scope.go:122] "RemoveContainer" containerID="4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657"
Oct 29 23:00:19.977166 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.977118    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:00:19.977264 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.977223    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:00:19.977325 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:19.977304    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:00:20.032612 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.032580    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"]
Oct 29 23:00:20.182197 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.180641    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:20.217286 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.215423    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:20.286486 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.285388    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:20.299777 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.299740    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930d7aea-d138-4786-ad20-850bc077cb9a-kube-api-access-j7zww\" (UniqueName: \"kubernetes.io/projected/930d7aea-d138-4786-ad20-850bc077cb9a-kube-api-access-j7zww\") pod \"930d7aea-d138-4786-ad20-850bc077cb9a\" (UID: \"930d7aea-d138-4786-ad20-850bc077cb9a\") "
Oct 29 23:00:20.300010 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.299838    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de665f9-a7b0-46db-8e52-eacd8612aa4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de665f9-a7b0-46db-8e52-eacd8612aa4d-maas-api-service-ca\") pod \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\" (UID: \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\") "
Oct 29 23:00:20.300010 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.299947    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de665f9-a7b0-46db-8e52-eacd8612aa4d-kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/8de665f9-a7b0-46db-8e52-eacd8612aa4d-kube-api-access-m5m75\") pod \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\" (UID: \"8de665f9-a7b0-46db-8e52-eacd8612aa4d\") "
Oct 29 23:00:20.300340 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.300103    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930d7aea-d138-4786-ad20-850bc077cb9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/930d7aea-d138-4786-ad20-850bc077cb9a-maas-api-service-ca\") pod \"930d7aea-d138-4786-ad20-850bc077cb9a\" (UID: \"930d7aea-d138-4786-ad20-850bc077cb9a\") "
Oct 29 23:00:20.301155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.301122    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930d7aea-d138-4786-ad20-850bc077cb9a-maas-api-service-ca" pod "930d7aea-d138-4786-ad20-850bc077cb9a" (UID: "930d7aea-d138-4786-ad20-850bc077cb9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.301533 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.301509    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de665f9-a7b0-46db-8e52-eacd8612aa4d-maas-api-service-ca" pod "8de665f9-a7b0-46db-8e52-eacd8612aa4d" (UID: "8de665f9-a7b0-46db-8e52-eacd8612aa4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.302814 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.302774    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930d7aea-d138-4786-ad20-850bc077cb9a-kube-api-access-j7zww" pod "930d7aea-d138-4786-ad20-850bc077cb9a" (UID: "930d7aea-d138-4786-ad20-850bc077cb9a"). InnerVolumeSpecName "kube-api-access-j7zww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.302956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.302884    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de665f9-a7b0-46db-8e52-eacd8612aa4d-kube-api-access-m5m75" pod "8de665f9-a7b0-46db-8e52-eacd8612aa4d" (UID: "8de665f9-a7b0-46db-8e52-eacd8612aa4d"). InnerVolumeSpecName "kube-api-access-m5m75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.303246 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.303219    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/930d7aea-d138-4786-ad20-850bc077cb9a-maas-api-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.303367 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.303250    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7zww\" (UniqueName: \"kubernetes.io/projected/930d7aea-d138-4786-ad20-850bc077cb9a-kube-api-access-j7zww\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.303367 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.303265    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de665f9-a7b0-46db-8e52-eacd8612aa4d-maas-api-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.303367 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.303279    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/8de665f9-a7b0-46db-8e52-eacd8612aa4d-kube-api-access-m5m75\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.404307 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.404272    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f181f5-a403-425a-83ff-f9eb52eed205-kube-api-access-r88d2\" (UniqueName: \"kubernetes.io/projected/e8f181f5-a403-425a-83ff-f9eb52eed205-kube-api-access-r88d2\") pod \"e8f181f5-a403-425a-83ff-f9eb52eed205\" (UID: \"e8f181f5-a403-425a-83ff-f9eb52eed205\") "
Oct 29 23:00:20.404511 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.404337    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f181f5-a403-425a-83ff-f9eb52eed205-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f181f5-a403-425a-83ff-f9eb52eed205-maas-api-service-ca\") pod \"e8f181f5-a403-425a-83ff-f9eb52eed205\" (UID: \"e8f181f5-a403-425a-83ff-f9eb52eed205\") "
Oct 29 23:00:20.404712 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.404682    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f181f5-a403-425a-83ff-f9eb52eed205-maas-api-service-ca" pod "e8f181f5-a403-425a-83ff-f9eb52eed205" (UID: "e8f181f5-a403-425a-83ff-f9eb52eed205"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.404978 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.404957    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f181f5-a403-425a-83ff-f9eb52eed205-maas-api-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.407099 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.407060    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f181f5-a403-425a-83ff-f9eb52eed205-kube-api-access-r88d2" pod "e8f181f5-a403-425a-83ff-f9eb52eed205" (UID: "e8f181f5-a403-425a-83ff-f9eb52eed205"). InnerVolumeSpecName "kube-api-access-r88d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.506495 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.506408    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r88d2\" (UniqueName: \"kubernetes.io/projected/e8f181f5-a403-425a-83ff-f9eb52eed205-kube-api-access-r88d2\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.543065 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.543020    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"]
Oct 29 23:00:20.544264 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.544236    4742 cpu_manager.go:414] "containerMap: removing container" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:20.544416 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.544403    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:20.544551 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.544536    4742 cpu_manager.go:414] "containerMap: removing container" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:20.544645 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.544630    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:20.544745 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.544732    4742 cpu_manager.go:414] "containerMap: removing container" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:20.544844 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.544829    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:20.544938 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.544928    4742 cpu_manager.go:414] "containerMap: removing container" podUID="ff7d5521-a843-4354-bbb0-c3d042da8f4e" containerName="payload-pre-processing"
Oct 29 23:00:20.545011 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545000    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7d5521-a843-4354-bbb0-c3d042da8f4e" containerName="payload-pre-processing"
Oct 29 23:00:20.545106 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545078    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:20.545197 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545186    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:20.545273 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545264    4742 cpu_manager.go:414] "containerMap: removing container" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:20.545345 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545335    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:20.545492 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545480    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:20.545570 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545560    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:20.545669 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545659    4742 cpu_manager.go:414] "containerMap: removing container" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:20.545742 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545732    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:20.545817 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545808    4742 cpu_manager.go:414] "containerMap: removing container" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:20.545892 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.545882    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:20.546209 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546188    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:20.546340 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546331    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:20.546425 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546416    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:20.546498 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546489    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:20.546571 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546563    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:20.546652 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546643    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:20.546728 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546719    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:20.546803 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546795    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:20.546876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546868    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:20.546948 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.546939    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7d5521-a843-4354-bbb0-c3d042da8f4e" containerName="payload-pre-processing"
Oct 29 23:00:20.547016 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.547008    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:20.551658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.551629    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.566114 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.564238    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"]
Oct 29 23:00:20.587280 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.575355    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:20.709182 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.709148    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.709331 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.709216    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.709367 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.709328    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c77sq\" (UniqueName: \"kubernetes.io/projected/1990d261-f5fb-4ade-94f9-1f15a3f3106f-kube-api-access-c77sq\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.810726 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.810640    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c77sq\" (UniqueName: \"kubernetes.io/projected/1990d261-f5fb-4ade-94f9-1f15a3f3106f-kube-api-access-c77sq\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.810881 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.810855    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.810931 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.810916    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.811525 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.811503    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.813367 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.813341    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.830690 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.830661    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c77sq\" (UniqueName: \"kubernetes.io/projected/1990d261-f5fb-4ade-94f9-1f15a3f3106f-kube-api-access-c77sq\") pod \"maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:20.895618 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:20.895589    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:21.001007 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.000716    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"
Oct 29 23:00:21.003104 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.003059    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"
Oct 29 23:00:21.004477 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.004454    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" path="/var/lib/kubelet/pods/be0273fd-b084-47ee-86e0-89fcb415793a/volumes"
Oct 29 23:00:21.005341 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.005316    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh" event={"ID":"8de665f9-a7b0-46db-8e52-eacd8612aa4d","Type":"ContainerDied","Data":"302b6c65bb2caf8ace8dd8735ac06bb0d8b7d683d14c0d40f1a6b7b33773c87d"}
Oct 29 23:00:21.005393 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.005375    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp" event={"ID":"930d7aea-d138-4786-ad20-850bc077cb9a","Type":"ContainerDied","Data":"5c5f9aa7c5e9f97acda0eeeec7ed156b41bc7781ca97b8f9dc0e466e7df49e54"}
Oct 29 23:00:21.005436 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.005409    4742 scope.go:122] "RemoveContainer" containerID="a60deea1332aec1fa490cb344562004b5b6fe9c4884c32563a72f62df33f516d"
Oct 29 23:00:21.010118 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.010059    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" event={"ID":"67aa34c6-25e3-4dc4-a57e-1dd90de4444c","Type":"ContainerStarted","Data":"d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5"}
Oct 29 23:00:21.019425 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.017212    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn" event={"ID":"e8f181f5-a403-425a-83ff-f9eb52eed205","Type":"ContainerDied","Data":"c2bd7b4af6cbaee57a6e22fd058a81e73d1e7f8135e5743b12264bcc4d629bab"}
Oct 29 23:00:21.019425 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.017297    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"
Oct 29 23:00:21.091290 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.090779    4742 scope.go:122] "RemoveContainer" containerID="764d23ca2a25ddefc7f944ab48c4bdc7c287a46bf2d72b9edff4e30437a56a5e"
Oct 29 23:00:21.107657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.107628    4742 scope.go:122] "RemoveContainer" containerID="ac0146360444484a19bde2d9aa3db879ad5db1d16d882213548ad204d373de52"
Oct 29 23:00:21.223552 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.223525    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"]
Oct 29 23:00:21.230125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.230070    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-20f88f-29888580-5dskh"]
Oct 29 23:00:21.239047 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.239027    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"]
Oct 29 23:00:21.290176 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.290146    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"]
Oct 29 23:00:21.292864 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.292845    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-hfcjp"]
Oct 29 23:00:21.306492 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.306472    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:21.319288 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.319256    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"]
Oct 29 23:00:21.319525 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.319491    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d" podUID="e13562bb-15c0-478d-9a25-2644bf11c412" containerName="maas-api" containerID="cri-o://75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2" gracePeriod=30
Oct 29 23:00:21.325043 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.325005    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"]
Oct 29 23:00:21.334971 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.334939    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-07eef2-29888580-r2ldn"]
Oct 29 23:00:21.427419 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.427382    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cb03fa-2ed7-4739-9536-901f5fece9c4-kube-api-access-dfk27\" (UniqueName: \"kubernetes.io/projected/e4cb03fa-2ed7-4739-9536-901f5fece9c4-kube-api-access-dfk27\") pod \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\" (UID: \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\") "
Oct 29 23:00:21.427604 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.427489    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cb03fa-2ed7-4739-9536-901f5fece9c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb03fa-2ed7-4739-9536-901f5fece9c4-maas-api-service-ca\") pod \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\" (UID: \"e4cb03fa-2ed7-4739-9536-901f5fece9c4\") "
Oct 29 23:00:21.427925 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.427898    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cb03fa-2ed7-4739-9536-901f5fece9c4-maas-api-service-ca" pod "e4cb03fa-2ed7-4739-9536-901f5fece9c4" (UID: "e4cb03fa-2ed7-4739-9536-901f5fece9c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:21.428265 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.428244    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cb03fa-2ed7-4739-9536-901f5fece9c4-maas-api-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.430960 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.430939    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 23:00:21.431186 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.431162    4742 patch_prober.go:29] interesting pod/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 23:00:21.431248 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.431223    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 23:00:21.491179 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.491146    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cb03fa-2ed7-4739-9536-901f5fece9c4-kube-api-access-dfk27" pod "e4cb03fa-2ed7-4739-9536-901f5fece9c4" (UID: "e4cb03fa-2ed7-4739-9536-901f5fece9c4"). InnerVolumeSpecName "kube-api-access-dfk27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:21.529111 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.529073    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfk27\" (UniqueName: \"kubernetes.io/projected/e4cb03fa-2ed7-4739-9536-901f5fece9c4-kube-api-access-dfk27\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.846107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.843278    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 23:00:21.859230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.859195    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-metrics-certs\") pod \"e13562bb-15c0-478d-9a25-2644bf11c412\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") "
Oct 29 23:00:21.859521 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.859499    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13562bb-15c0-478d-9a25-2644bf11c412-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13562bb-15c0-478d-9a25-2644bf11c412-openshift-service-ca\") pod \"e13562bb-15c0-478d-9a25-2644bf11c412\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") "
Oct 29 23:00:21.859718 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.859704    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13562bb-15c0-478d-9a25-2644bf11c412-kube-api-access-dc7hw\" (UniqueName: \"kubernetes.io/projected/e13562bb-15c0-478d-9a25-2644bf11c412-kube-api-access-dc7hw\") pod \"e13562bb-15c0-478d-9a25-2644bf11c412\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") "
Oct 29 23:00:21.859829 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.859815    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls\") pod \"e13562bb-15c0-478d-9a25-2644bf11c412\" (UID: \"e13562bb-15c0-478d-9a25-2644bf11c412\") "
Oct 29 23:00:21.875614 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.871646    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls" pod "e13562bb-15c0-478d-9a25-2644bf11c412" (UID: "e13562bb-15c0-478d-9a25-2644bf11c412"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:21.883318 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.882199    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13562bb-15c0-478d-9a25-2644bf11c412-kube-api-access-dc7hw" pod "e13562bb-15c0-478d-9a25-2644bf11c412" (UID: "e13562bb-15c0-478d-9a25-2644bf11c412"). InnerVolumeSpecName "kube-api-access-dc7hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:21.886244 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.886205    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13562bb-15c0-478d-9a25-2644bf11c412-openshift-service-ca" pod "e13562bb-15c0-478d-9a25-2644bf11c412" (UID: "e13562bb-15c0-478d-9a25-2644bf11c412"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:21.892297 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.892261    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-metrics-certs" pod "e13562bb-15c0-478d-9a25-2644bf11c412" (UID: "e13562bb-15c0-478d-9a25-2644bf11c412"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:21.971172 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:21.967716    4742 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13562bb-15c0-478d-9a25-2644bf11c412-openshift-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.003123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.001466    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc7hw\" (UniqueName: \"kubernetes.io/projected/e13562bb-15c0-478d-9a25-2644bf11c412-kube-api-access-dc7hw\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.003123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.001501    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-maas-api-tls\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.003123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.001518    4742 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13562bb-15c0-478d-9a25-2644bf11c412-metrics-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.037299 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.037268    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p" event={"ID":"e4cb03fa-2ed7-4739-9536-901f5fece9c4","Type":"ContainerDied","Data":"e4bd634d0041cdb7146ff74c7b3dae7a59b594791cf58679513a322be73eea34"}
Oct 29 23:00:22.037876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.037618    4742 scope.go:122] "RemoveContainer" containerID="86485440481ef79c39c0c1b75a8b627f7b0b91ad0e415e9ec02d050ca90b76a6"
Oct 29 23:00:22.037876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.037535    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"
Oct 29 23:00:22.038925 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.038906    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"]
Oct 29 23:00:22.061107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.046786    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1291a8-jd8qs"]
Oct 29 23:00:22.081657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.078913    4742 generic.go:356] "Generic (PLEG): container finished" podID="e13562bb-15c0-478d-9a25-2644bf11c412" containerID="75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2" exitCode=0
Oct 29 23:00:22.081657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.079048    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d" event={"ID":"e13562bb-15c0-478d-9a25-2644bf11c412","Type":"ContainerDied","Data":"75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2"}
Oct 29 23:00:22.081657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.079110    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d" event={"ID":"e13562bb-15c0-478d-9a25-2644bf11c412","Type":"ContainerDied","Data":"9d37908f933d7e76b8e84491f074d6f3d8db958ead132ff0850f0399a5dc133e"}
Oct 29 23:00:22.081657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.079198    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"
Oct 29 23:00:22.090375 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.090193    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj" event={"ID":"1990d261-f5fb-4ade-94f9-1f15a3f3106f","Type":"ContainerStarted","Data":"a6d5a873cb2fb51dc85a38d22f10568eee911517026a76f597f3320e96f22648"}
Oct 29 23:00:22.090375 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.090265    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj" event={"ID":"1990d261-f5fb-4ade-94f9-1f15a3f3106f","Type":"ContainerStarted","Data":"766c379b53563f47dbd49549977bd3b1148a4fa05475b668dee0803bcc60b58f"}
Oct 29 23:00:22.094180 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.094104    4742 scope.go:122] "RemoveContainer" containerID="75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2"
Oct 29 23:00:22.105695 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.105677    4742 scope.go:122] "RemoveContainer" containerID="75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2"
Oct 29 23:00:22.105962 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:22.105943    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2\": container with ID starting with 75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2 not found: ID does not exist" containerID="75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2"
Oct 29 23:00:22.106008 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.105971    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2"} err="failed to get container status \"75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2\": rpc error: code = NotFound desc = could not find container \"75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2\": container with ID starting with 75b67dd4ea36f04b5d1da5785cb8bce6bcb6f31290ed5e090e1a38dccdb936c2 not found: ID does not exist"
Oct 29 23:00:22.134635 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.134582    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj" podStartSLOduration=2.1345686600000002 podStartE2EDuration="2.13456866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:22.130886814 +0000 UTC m=+1079.271628412" watchObservedRunningTime="2026-10-29 23:00:22.13456866 +0000 UTC m=+1079.275310243"
Oct 29 23:00:22.190400 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.190367    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"]
Oct 29 23:00:22.194207 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.194182    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-81f92d-29888580-5sw2p"]
Oct 29 23:00:22.247256 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.247177    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"]
Oct 29 23:00:22.250908 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.250884    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34d0fd42-94cd7bc7f-gz49d"]
Oct 29 23:00:22.430454 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.430422    4742 patch_prober.go:29] interesting pod/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 23:00:22.430653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.430473    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 23:00:22.994123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.994027    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077d0086-224d-4657-994b-5cbbafe2407c" path="/var/lib/kubelet/pods/077d0086-224d-4657-994b-5cbbafe2407c/volumes"
Oct 29 23:00:22.994462 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.994448    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" path="/var/lib/kubelet/pods/8de665f9-a7b0-46db-8e52-eacd8612aa4d/volumes"
Oct 29 23:00:22.994831 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.994819    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" path="/var/lib/kubelet/pods/930d7aea-d138-4786-ad20-850bc077cb9a/volumes"
Oct 29 23:00:22.995468 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.995450    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13562bb-15c0-478d-9a25-2644bf11c412" path="/var/lib/kubelet/pods/e13562bb-15c0-478d-9a25-2644bf11c412/volumes"
Oct 29 23:00:22.995819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.995806    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" path="/var/lib/kubelet/pods/e4cb03fa-2ed7-4739-9536-901f5fece9c4/volumes"
Oct 29 23:00:22.996200 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:22.996186    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" path="/var/lib/kubelet/pods/e8f181f5-a403-425a-83ff-f9eb52eed205/volumes"
Oct 29 23:00:23.101356 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:23.101324    4742 generic.go:356] "Generic (PLEG): container finished" podID="1990d261-f5fb-4ade-94f9-1f15a3f3106f" containerID="a6d5a873cb2fb51dc85a38d22f10568eee911517026a76f597f3320e96f22648" exitCode=0
Oct 29 23:00:23.101820 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:23.101404    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj" event={"ID":"1990d261-f5fb-4ade-94f9-1f15a3f3106f","Type":"ContainerDied","Data":"a6d5a873cb2fb51dc85a38d22f10568eee911517026a76f597f3320e96f22648"}
Oct 29 23:00:23.125473 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:23.125436    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:00:23.431149 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:23.431114    4742 patch_prober.go:29] interesting pod/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 23:00:23.431320 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:23.431167    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 23:00:24.403072 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.402991    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:24.431063 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.431027    4742 patch_prober.go:29] interesting pod/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 23:00:24.431401 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.431123    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 23:00:24.450309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.448695    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-openshift-ca\") pod \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") "
Oct 29 23:00:24.450309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.448755    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1990d261-f5fb-4ade-94f9-1f15a3f3106f-kube-api-access-c77sq\" (UniqueName: \"kubernetes.io/projected/1990d261-f5fb-4ade-94f9-1f15a3f3106f-kube-api-access-c77sq\") pod \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") "
Oct 29 23:00:24.450309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.448797    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-xks-ca\") pod \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\" (UID: \"1990d261-f5fb-4ade-94f9-1f15a3f3106f\") "
Oct 29 23:00:24.450309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.449946    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-openshift-ca" pod "1990d261-f5fb-4ade-94f9-1f15a3f3106f" (UID: "1990d261-f5fb-4ade-94f9-1f15a3f3106f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.452230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.451902    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1990d261-f5fb-4ade-94f9-1f15a3f3106f-kube-api-access-c77sq" pod "1990d261-f5fb-4ade-94f9-1f15a3f3106f" (UID: "1990d261-f5fb-4ade-94f9-1f15a3f3106f"). InnerVolumeSpecName "kube-api-access-c77sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.454477 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.454452    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-xks-ca" pod "1990d261-f5fb-4ade-94f9-1f15a3f3106f" (UID: "1990d261-f5fb-4ade-94f9-1f15a3f3106f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:24.550518 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.550477    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c77sq\" (UniqueName: \"kubernetes.io/projected/1990d261-f5fb-4ade-94f9-1f15a3f3106f-kube-api-access-c77sq\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.550518 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.550511    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.550518 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.550525    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1990d261-f5fb-4ade-94f9-1f15a3f3106f-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.587215 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587178    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"]
Oct 29 23:00:24.587880 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587859    4742 cpu_manager.go:414] "containerMap: removing container" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:24.587979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587882    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:24.587979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587904    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:24.587979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587911    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:24.587979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587939    4742 cpu_manager.go:414] "containerMap: removing container" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:24.587979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587947    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930d7aea-d138-4786-ad20-850bc077cb9a" containerName="cleanup"
Oct 29 23:00:24.587979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587968    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e13562bb-15c0-478d-9a25-2644bf11c412" containerName="maas-api"
Oct 29 23:00:24.587979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587975    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13562bb-15c0-478d-9a25-2644bf11c412" containerName="maas-api"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587990    4742 cpu_manager.go:414] "containerMap: removing container" podUID="1990d261-f5fb-4ade-94f9-1f15a3f3106f" containerName="revoke-keys"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.587998    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1990d261-f5fb-4ade-94f9-1f15a3f3106f" containerName="revoke-keys"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588019    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588027    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588041    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588048    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588074    4742 cpu_manager.go:414] "containerMap: removing container" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588122    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de665f9-a7b0-46db-8e52-eacd8612aa4d" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588273    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13562bb-15c0-478d-9a25-2644bf11c412" containerName="maas-api"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588288    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588297    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="1990d261-f5fb-4ade-94f9-1f15a3f3106f" containerName="revoke-keys"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588307    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f181f5-a403-425a-83ff-f9eb52eed205" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588320    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0273fd-b084-47ee-86e0-89fcb415793a" containerName="cleanup"
Oct 29 23:00:24.588346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.588334    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:24.592276 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.592256    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.608118 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.608058    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"]
Oct 29 23:00:24.651598 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.651558    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tssn\" (UniqueName: \"kubernetes.io/projected/3fd2e0d2-03d4-428c-af26-144dab9d04ca-kube-api-access-4tssn\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.651857 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.651832    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.652012 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.651896    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.753269 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.753174    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tssn\" (UniqueName: \"kubernetes.io/projected/3fd2e0d2-03d4-428c-af26-144dab9d04ca-kube-api-access-4tssn\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.753431 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.753299    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.753431 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.753330    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.754098 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.754056    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.755658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.755640    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.786451 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.786415    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tssn\" (UniqueName: \"kubernetes.io/projected/3fd2e0d2-03d4-428c-af26-144dab9d04ca-kube-api-access-4tssn\") pod \"maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:24.910869 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:24.910836    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:25.136473 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:25.136440    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"
Oct 29 23:00:25.136473 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:25.136464    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj" event={"ID":"1990d261-f5fb-4ade-94f9-1f15a3f3106f","Type":"ContainerDied","Data":"766c379b53563f47dbd49549977bd3b1148a4fa05475b668dee0803bcc60b58f"}
Oct 29 23:00:25.136708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:25.136516    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="766c379b53563f47dbd49549977bd3b1148a4fa05475b668dee0803bcc60b58f"
Oct 29 23:00:25.241255 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:25.241225    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"]
Oct 29 23:00:25.241255 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:25.241236    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd2e0d2_03d4_428c_af26_144dab9d04ca.slice/crio-3687b1e60e045e8f275cc3da8a8b5d4de94cbb3be5b9ec2aeb941ebdd829b053 WatchSource:0}: Error finding container 3687b1e60e045e8f275cc3da8a8b5d4de94cbb3be5b9ec2aeb941ebdd829b053: Status 404 returned error can't find the container with id 3687b1e60e045e8f275cc3da8a8b5d4de94cbb3be5b9ec2aeb941ebdd829b053
Oct 29 23:00:25.431232 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:25.431198    4742 patch_prober.go:29] interesting pod/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 29 23:00:25.431588 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:25.431265    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 29 23:00:26.145436 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.145399    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf" event={"ID":"3fd2e0d2-03d4-428c-af26-144dab9d04ca","Type":"ContainerStarted","Data":"1ecdbe2acd710b6a7c47928d1a845a26385017374766426f84b7714d84d08e5e"}
Oct 29 23:00:26.145635 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.145460    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf" event={"ID":"3fd2e0d2-03d4-428c-af26-144dab9d04ca","Type":"ContainerStarted","Data":"3687b1e60e045e8f275cc3da8a8b5d4de94cbb3be5b9ec2aeb941ebdd829b053"}
Oct 29 23:00:26.209272 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.209210    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf" podStartSLOduration=2.20919124 podStartE2EDuration="2.20919124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:26.2050707 +0000 UTC m=+1083.345812286" watchObservedRunningTime="2026-10-29 23:00:26.20919124 +0000 UTC m=+1083.349932859"
Oct 29 23:00:26.434972 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.434899    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 23:00:26.435448 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.435213    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 23:00:26.435927 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.435912    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 23:00:26.649559 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.649523    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"]
Oct 29 23:00:26.650597 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.650559    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:26.650597 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.650586    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:26.650898 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.650732    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cb03fa-2ed7-4739-9536-901f5fece9c4" containerName="cleanup"
Oct 29 23:00:26.655129 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.654957    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.670329 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.670305    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"]
Oct 29 23:00:26.806682 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.806601    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.806682 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.806636    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8dlq\" (UniqueName: \"kubernetes.io/projected/d85c4b25-31c7-4246-b7d7-3fecbff10238-kube-api-access-j8dlq\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.806906 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.806869    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.907753 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.907716    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.907899 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.907840    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.907899 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.907865    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dlq\" (UniqueName: \"kubernetes.io/projected/d85c4b25-31c7-4246-b7d7-3fecbff10238-kube-api-access-j8dlq\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.908566 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.908540    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.910266 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.910238    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.926353 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.926327    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dlq\" (UniqueName: \"kubernetes.io/projected/d85c4b25-31c7-4246-b7d7-3fecbff10238-kube-api-access-j8dlq\") pod \"maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:26.970004 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:26.969970    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:27.153796 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:27.153757    4742 generic.go:356] "Generic (PLEG): container finished" podID="3fd2e0d2-03d4-428c-af26-144dab9d04ca" containerID="1ecdbe2acd710b6a7c47928d1a845a26385017374766426f84b7714d84d08e5e" exitCode=0
Oct 29 23:00:27.154003 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:27.153844    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf" event={"ID":"3fd2e0d2-03d4-428c-af26-144dab9d04ca","Type":"ContainerDied","Data":"1ecdbe2acd710b6a7c47928d1a845a26385017374766426f84b7714d84d08e5e"}
Oct 29 23:00:27.271283 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:27.271255    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"]
Oct 29 23:00:27.271848 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:27.271826    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85c4b25_31c7_4246_b7d7_3fecbff10238.slice/crio-5649874f4f404de3133acc4c1793dcdae035b9f186247af3d3b9195b25aacaf3 WatchSource:0}: Error finding container 5649874f4f404de3133acc4c1793dcdae035b9f186247af3d3b9195b25aacaf3: Status 404 returned error can't find the container with id 5649874f4f404de3133acc4c1793dcdae035b9f186247af3d3b9195b25aacaf3
Oct 29 23:00:28.171279 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.171241    4742 generic.go:356] "Generic (PLEG): container finished" podID="d85c4b25-31c7-4246-b7d7-3fecbff10238" containerID="d169e44df1bc43d327a71f247b77b613098dd12bc5f79bd8c705ae5f6db5b38f" exitCode=0
Oct 29 23:00:28.171889 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.171336    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl" event={"ID":"d85c4b25-31c7-4246-b7d7-3fecbff10238","Type":"ContainerDied","Data":"d169e44df1bc43d327a71f247b77b613098dd12bc5f79bd8c705ae5f6db5b38f"}
Oct 29 23:00:28.171889 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.171829    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl" event={"ID":"d85c4b25-31c7-4246-b7d7-3fecbff10238","Type":"ContainerStarted","Data":"5649874f4f404de3133acc4c1793dcdae035b9f186247af3d3b9195b25aacaf3"}
Oct 29 23:00:28.359288 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.359227    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:28.427192 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.427161    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-xks-ca\") pod \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") "
Oct 29 23:00:28.427391 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.427263    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd2e0d2-03d4-428c-af26-144dab9d04ca-kube-api-access-4tssn\" (UniqueName: \"kubernetes.io/projected/3fd2e0d2-03d4-428c-af26-144dab9d04ca-kube-api-access-4tssn\") pod \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") "
Oct 29 23:00:28.427391 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.427382    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-openshift-ca\") pod \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\" (UID: \"3fd2e0d2-03d4-428c-af26-144dab9d04ca\") "
Oct 29 23:00:28.427827 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.427792    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-openshift-ca" pod "3fd2e0d2-03d4-428c-af26-144dab9d04ca" (UID: "3fd2e0d2-03d4-428c-af26-144dab9d04ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:28.428160 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.428135    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:28.429518 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.429489    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd2e0d2-03d4-428c-af26-144dab9d04ca-kube-api-access-4tssn" pod "3fd2e0d2-03d4-428c-af26-144dab9d04ca" (UID: "3fd2e0d2-03d4-428c-af26-144dab9d04ca"). InnerVolumeSpecName "kube-api-access-4tssn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:28.429779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.429752    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-xks-ca" pod "3fd2e0d2-03d4-428c-af26-144dab9d04ca" (UID: "3fd2e0d2-03d4-428c-af26-144dab9d04ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:28.529317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.529282    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd2e0d2-03d4-428c-af26-144dab9d04ca-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:28.529317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.529311    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tssn\" (UniqueName: \"kubernetes.io/projected/3fd2e0d2-03d4-428c-af26-144dab9d04ca-kube-api-access-4tssn\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:28.992462 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:28.992433    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"]
Oct 29 23:00:29.180345 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.180319    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"
Oct 29 23:00:29.180729 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.180362    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf" event={"ID":"3fd2e0d2-03d4-428c-af26-144dab9d04ca","Type":"ContainerDied","Data":"3687b1e60e045e8f275cc3da8a8b5d4de94cbb3be5b9ec2aeb941ebdd829b053"}
Oct 29 23:00:29.180729 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.180417    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3687b1e60e045e8f275cc3da8a8b5d4de94cbb3be5b9ec2aeb941ebdd829b053"
Oct 29 23:00:29.379036 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.378971    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:29.386467 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.386437    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"]
Oct 29 23:00:29.449245 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.449213    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85c4b25-31c7-4246-b7d7-3fecbff10238-kube-api-access-j8dlq\" (UniqueName: \"kubernetes.io/projected/d85c4b25-31c7-4246-b7d7-3fecbff10238-kube-api-access-j8dlq\") pod \"d85c4b25-31c7-4246-b7d7-3fecbff10238\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") "
Oct 29 23:00:29.449434 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.449345    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-xks-ca\") pod \"d85c4b25-31c7-4246-b7d7-3fecbff10238\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") "
Oct 29 23:00:29.449516 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.449473    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-openshift-ca\") pod \"d85c4b25-31c7-4246-b7d7-3fecbff10238\" (UID: \"d85c4b25-31c7-4246-b7d7-3fecbff10238\") "
Oct 29 23:00:29.449870 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.449838    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-openshift-ca" pod "d85c4b25-31c7-4246-b7d7-3fecbff10238" (UID: "d85c4b25-31c7-4246-b7d7-3fecbff10238"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:29.450026 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.449984    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.451548 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.451520    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85c4b25-31c7-4246-b7d7-3fecbff10238-kube-api-access-j8dlq" pod "d85c4b25-31c7-4246-b7d7-3fecbff10238" (UID: "d85c4b25-31c7-4246-b7d7-3fecbff10238"). InnerVolumeSpecName "kube-api-access-j8dlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:29.451931 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.451910    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-xks-ca" pod "d85c4b25-31c7-4246-b7d7-3fecbff10238" (UID: "d85c4b25-31c7-4246-b7d7-3fecbff10238"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:29.551903 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.551818    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8dlq\" (UniqueName: \"kubernetes.io/projected/d85c4b25-31c7-4246-b7d7-3fecbff10238-kube-api-access-j8dlq\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.551903 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:29.551851    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c4b25-31c7-4246-b7d7-3fecbff10238-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:30.190775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:30.190743    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"
Oct 29 23:00:30.190775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:30.190766    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl" event={"ID":"d85c4b25-31c7-4246-b7d7-3fecbff10238","Type":"ContainerDied","Data":"5649874f4f404de3133acc4c1793dcdae035b9f186247af3d3b9195b25aacaf3"}
Oct 29 23:00:30.191359 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:30.190822    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5649874f4f404de3133acc4c1793dcdae035b9f186247af3d3b9195b25aacaf3"
Oct 29 23:00:30.966448 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:30.966409    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"]
Oct 29 23:00:31.007136 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:31.007097    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"]
Oct 29 23:00:31.449338 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:31.449308    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"]
Oct 29 23:00:31.466407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:31.466375    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"]
Oct 29 23:00:31.587617 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:31.587579    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"]
Oct 29 23:00:33.992949 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:33.992904    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm" podUID="fb308604-255e-4f3e-8c89-8b9b571e19da" containerName="payload-processing" containerID="cri-o://f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf" gracePeriod=25
Oct 29 23:00:34.742614 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.742578    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"]
Oct 29 23:00:34.743181 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.743166    4742 cpu_manager.go:414] "containerMap: removing container" podUID="3fd2e0d2-03d4-428c-af26-144dab9d04ca" containerName="revoke-keys"
Oct 29 23:00:34.743231 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.743183    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd2e0d2-03d4-428c-af26-144dab9d04ca" containerName="revoke-keys"
Oct 29 23:00:34.743231 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.743206    4742 cpu_manager.go:414] "containerMap: removing container" podUID="d85c4b25-31c7-4246-b7d7-3fecbff10238" containerName="revoke-keys"
Oct 29 23:00:34.743231 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.743211    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85c4b25-31c7-4246-b7d7-3fecbff10238" containerName="revoke-keys"
Oct 29 23:00:34.743323 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.743316    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd2e0d2-03d4-428c-af26-144dab9d04ca" containerName="revoke-keys"
Oct 29 23:00:34.743361 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.743328    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85c4b25-31c7-4246-b7d7-3fecbff10238" containerName="revoke-keys"
Oct 29 23:00:34.747527 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.747506    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.764283 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.764241    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"]
Oct 29 23:00:34.768791 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.768768    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.770724 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.770697    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"]
Oct 29 23:00:34.771506 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.771488    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:34.771625 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.771606    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:00:34.797981 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.797952    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"]
Oct 29 23:00:34.851555 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.851519    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.851724 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.851634    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.851789 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.851775    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.851836 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.851824    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.851909 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.851893    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gsh\" (UniqueName: \"kubernetes.io/projected/6e73e4ee-008e-43bd-867d-106f11547d5a-kube-api-access-l5gsh\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.851962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.851951    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-kube-api-access-8rcrc\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.953591 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.953555    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-kube-api-access-8rcrc\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.953591 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.953596    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.953790 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.953652    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.953790 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.953735    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.953889 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.953868    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.953948 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.953929    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gsh\" (UniqueName: \"kubernetes.io/projected/6e73e4ee-008e-43bd-867d-106f11547d5a-kube-api-access-l5gsh\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.954477 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.954452    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.954589 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.954496    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.956580 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.956558    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.956580 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.956566    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:34.974237 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.974211    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-kube-api-access-8rcrc\") pod \"maas-api-revoke-sub-9edb9646-m4v49\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:34.974435 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:34.974416    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gsh\" (UniqueName: \"kubernetes.io/projected/6e73e4ee-008e-43bd-867d-106f11547d5a-kube-api-access-l5gsh\") pod \"maas-api-revoke-sub-67c1321f-84nz5\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:35.062702 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:35.062595    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:35.091093 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:35.091050    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:35.197872 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:35.197625    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm" podUID="fb308604-255e-4f3e-8c89-8b9b571e19da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:35.396571 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:35.396530    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15f3912_65fe_4d9f_8b45_3da75b4c56d3.slice/crio-48dc8ff956c02dea960fd0877cb11e63e90e4f07c64cb0587364560627a8df71 WatchSource:0}: Error finding container 48dc8ff956c02dea960fd0877cb11e63e90e4f07c64cb0587364560627a8df71: Status 404 returned error can't find the container with id 48dc8ff956c02dea960fd0877cb11e63e90e4f07c64cb0587364560627a8df71
Oct 29 23:00:35.396867 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:35.396845    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"]
Oct 29 23:00:35.452532 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:35.452499    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"]
Oct 29 23:00:35.967411 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:35.967372    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9" podUID="a95dd232-389e-456d-accb-65be45758b49" containerName="payload-processing" containerID="cri-o://59e910364a985b1e3bacf651a79f6df74afe178917ab6cdea8c30630ed01d68c" gracePeriod=25
Oct 29 23:00:36.007778 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.007738    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh" podUID="bc8cfc30-4859-4b97-8375-aa2ec37a009f" containerName="payload-pre-processing" containerID="cri-o://24facfedb2b5a9277ca427a9a8a08c0e2df0ea3f40553191ac5f7dc772cf0f6d" gracePeriod=25
Oct 29 23:00:36.253792 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.253763    4742 generic.go:356] "Generic (PLEG): container finished" podID="a95dd232-389e-456d-accb-65be45758b49" containerID="59e910364a985b1e3bacf651a79f6df74afe178917ab6cdea8c30630ed01d68c" exitCode=0
Oct 29 23:00:36.254203 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.253859    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9" event={"ID":"a95dd232-389e-456d-accb-65be45758b49","Type":"ContainerDied","Data":"59e910364a985b1e3bacf651a79f6df74afe178917ab6cdea8c30630ed01d68c"}
Oct 29 23:00:36.256126 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.256061    4742 generic.go:356] "Generic (PLEG): container finished" podID="bc8cfc30-4859-4b97-8375-aa2ec37a009f" containerID="24facfedb2b5a9277ca427a9a8a08c0e2df0ea3f40553191ac5f7dc772cf0f6d" exitCode=0
Oct 29 23:00:36.256365 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.256345    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh" event={"ID":"bc8cfc30-4859-4b97-8375-aa2ec37a009f","Type":"ContainerDied","Data":"24facfedb2b5a9277ca427a9a8a08c0e2df0ea3f40553191ac5f7dc772cf0f6d"}
Oct 29 23:00:36.258815 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.258515    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5" event={"ID":"6e73e4ee-008e-43bd-867d-106f11547d5a","Type":"ContainerStarted","Data":"f9917f26e3b1e1144a0c27a6d22e476cd9fee94b983134b5dd3687f09306f86a"}
Oct 29 23:00:36.258815 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.258553    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5" event={"ID":"6e73e4ee-008e-43bd-867d-106f11547d5a","Type":"ContainerStarted","Data":"cad629d2e1c7b6f9a0a20d494acef622f4f30f86b3c540f8235ee8e6a7c0a7f0"}
Oct 29 23:00:36.263343 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.262220    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49" event={"ID":"a15f3912-65fe-4d9f-8b45-3da75b4c56d3","Type":"ContainerStarted","Data":"59d199dd17337f2e6a84c15d5472d20a15148f23db1ea1410ebd05f39b3bee19"}
Oct 29 23:00:36.263343 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.262261    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49" event={"ID":"a15f3912-65fe-4d9f-8b45-3da75b4c56d3","Type":"ContainerStarted","Data":"48dc8ff956c02dea960fd0877cb11e63e90e4f07c64cb0587364560627a8df71"}
Oct 29 23:00:36.288342 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.288316    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 23:00:36.321841 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.321790    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5" podStartSLOduration=2.32177477 podStartE2EDuration="2.32177477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:36.317463242 +0000 UTC m=+1093.458204826" watchObservedRunningTime="2026-10-29 23:00:36.32177477 +0000 UTC m=+1093.462516354"
Oct 29 23:00:36.326612 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.326591    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 23:00:36.361850 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.361787    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49" podStartSLOduration=2.361766045 podStartE2EDuration="2.361766045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:36.358283521 +0000 UTC m=+1093.499025115" watchObservedRunningTime="2026-10-29 23:00:36.361766045 +0000 UTC m=+1093.502507635"
Oct 29 23:00:36.398641 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.398611    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8cfc30-4859-4b97-8375-aa2ec37a009f-kube-api-access-2hxk5\" (UniqueName: \"kubernetes.io/projected/bc8cfc30-4859-4b97-8375-aa2ec37a009f-kube-api-access-2hxk5\") pod \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\" (UID: \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\") "
Oct 29 23:00:36.398996 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.398657    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8cfc30-4859-4b97-8375-aa2ec37a009f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8cfc30-4859-4b97-8375-aa2ec37a009f-plugins-config-volume\") pod \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\" (UID: \"bc8cfc30-4859-4b97-8375-aa2ec37a009f\") "
Oct 29 23:00:36.398996 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.398945    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95dd232-389e-456d-accb-65be45758b49-kube-api-access-bwrz6\" (UniqueName: \"kubernetes.io/projected/a95dd232-389e-456d-accb-65be45758b49-kube-api-access-bwrz6\") pod \"a95dd232-389e-456d-accb-65be45758b49\" (UID: \"a95dd232-389e-456d-accb-65be45758b49\") "
Oct 29 23:00:36.399136 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.399108    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95dd232-389e-456d-accb-65be45758b49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95dd232-389e-456d-accb-65be45758b49-plugins-config-volume\") pod \"a95dd232-389e-456d-accb-65be45758b49\" (UID: \"a95dd232-389e-456d-accb-65be45758b49\") "
Oct 29 23:00:36.399717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.399482    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8cfc30-4859-4b97-8375-aa2ec37a009f-plugins-config-volume" pod "bc8cfc30-4859-4b97-8375-aa2ec37a009f" (UID: "bc8cfc30-4859-4b97-8375-aa2ec37a009f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:36.399717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.399564    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95dd232-389e-456d-accb-65be45758b49-plugins-config-volume" pod "a95dd232-389e-456d-accb-65be45758b49" (UID: "a95dd232-389e-456d-accb-65be45758b49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:36.399905 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.399847    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95dd232-389e-456d-accb-65be45758b49-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.399905 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.399868    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8cfc30-4859-4b97-8375-aa2ec37a009f-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.401735 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.401708    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95dd232-389e-456d-accb-65be45758b49-kube-api-access-bwrz6" pod "a95dd232-389e-456d-accb-65be45758b49" (UID: "a95dd232-389e-456d-accb-65be45758b49"). InnerVolumeSpecName "kube-api-access-bwrz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:36.402242 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.402221    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8cfc30-4859-4b97-8375-aa2ec37a009f-kube-api-access-2hxk5" pod "bc8cfc30-4859-4b97-8375-aa2ec37a009f" (UID: "bc8cfc30-4859-4b97-8375-aa2ec37a009f"). InnerVolumeSpecName "kube-api-access-2hxk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:36.500876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.500813    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwrz6\" (UniqueName: \"kubernetes.io/projected/a95dd232-389e-456d-accb-65be45758b49-kube-api-access-bwrz6\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:36.500876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:36.500839    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxk5\" (UniqueName: \"kubernetes.io/projected/bc8cfc30-4859-4b97-8375-aa2ec37a009f-kube-api-access-2hxk5\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:37.112359 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.112324    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"]
Oct 29 23:00:37.270921 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.270827    4742 generic.go:356] "Generic (PLEG): container finished" podID="a15f3912-65fe-4d9f-8b45-3da75b4c56d3" containerID="59d199dd17337f2e6a84c15d5472d20a15148f23db1ea1410ebd05f39b3bee19" exitCode=0
Oct 29 23:00:37.270921 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.270911    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49" event={"ID":"a15f3912-65fe-4d9f-8b45-3da75b4c56d3","Type":"ContainerDied","Data":"59d199dd17337f2e6a84c15d5472d20a15148f23db1ea1410ebd05f39b3bee19"}
Oct 29 23:00:37.272340 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.272323    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"
Oct 29 23:00:37.272340 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.272329    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9" event={"ID":"a95dd232-389e-456d-accb-65be45758b49","Type":"ContainerDied","Data":"43b9e15fe11465d2a3a6de70b669f2a0132f4ca046bdd4cda072c480dd4050b7"}
Oct 29 23:00:37.272499 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.272369    4742 scope.go:122] "RemoveContainer" containerID="59e910364a985b1e3bacf651a79f6df74afe178917ab6cdea8c30630ed01d68c"
Oct 29 23:00:37.273921 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.273801    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"
Oct 29 23:00:37.273921 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.273826    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh" event={"ID":"bc8cfc30-4859-4b97-8375-aa2ec37a009f","Type":"ContainerDied","Data":"f5eda2ae8d25269835931e2b6fbac88055a7d7272f0365306b61fd2cbda99e94"}
Oct 29 23:00:37.275846 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.275817    4742 generic.go:356] "Generic (PLEG): container finished" podID="6e73e4ee-008e-43bd-867d-106f11547d5a" containerID="f9917f26e3b1e1144a0c27a6d22e476cd9fee94b983134b5dd3687f09306f86a" exitCode=0
Oct 29 23:00:37.275954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.275886    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5" event={"ID":"6e73e4ee-008e-43bd-867d-106f11547d5a","Type":"ContainerDied","Data":"f9917f26e3b1e1144a0c27a6d22e476cd9fee94b983134b5dd3687f09306f86a"}
Oct 29 23:00:37.283701 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.283678    4742 scope.go:122] "RemoveContainer" containerID="24facfedb2b5a9277ca427a9a8a08c0e2df0ea3f40553191ac5f7dc772cf0f6d"
Oct 29 23:00:37.392195 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.392152    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"]
Oct 29 23:00:37.398025 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.397990    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-28ed1cea-78cd77d695-jcjn9"]
Oct 29 23:00:37.437438 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.437410    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"]
Oct 29 23:00:37.444132 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.444074    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-28ed1cea-79d596c4f9-ngxhh"]
Oct 29 23:00:37.868745 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:37.868711    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"]
Oct 29 23:00:38.447538 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.447510    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:38.481890 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.481864    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:38.533099 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.533043    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e73e4ee-008e-43bd-867d-106f11547d5a-kube-api-access-l5gsh\" (UniqueName: \"kubernetes.io/projected/6e73e4ee-008e-43bd-867d-106f11547d5a-kube-api-access-l5gsh\") pod \"6e73e4ee-008e-43bd-867d-106f11547d5a\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") "
Oct 29 23:00:38.533292 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.533137    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-kube-api-access-8rcrc\") pod \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") "
Oct 29 23:00:38.533292 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.533223    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-xks-ca\") pod \"6e73e4ee-008e-43bd-867d-106f11547d5a\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") "
Oct 29 23:00:38.533411 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.533346    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-openshift-ca\") pod \"6e73e4ee-008e-43bd-867d-106f11547d5a\" (UID: \"6e73e4ee-008e-43bd-867d-106f11547d5a\") "
Oct 29 23:00:38.533471 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.533415    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-xks-ca\") pod \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") "
Oct 29 23:00:38.533471 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.533466    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-openshift-ca\") pod \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\" (UID: \"a15f3912-65fe-4d9f-8b45-3da75b4c56d3\") "
Oct 29 23:00:38.535032 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.534997    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-openshift-ca" pod "a15f3912-65fe-4d9f-8b45-3da75b4c56d3" (UID: "a15f3912-65fe-4d9f-8b45-3da75b4c56d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:38.608835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.596395    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-openshift-ca" pod "6e73e4ee-008e-43bd-867d-106f11547d5a" (UID: "6e73e4ee-008e-43bd-867d-106f11547d5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:38.608835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.596612    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e73e4ee-008e-43bd-867d-106f11547d5a-kube-api-access-l5gsh" pod "6e73e4ee-008e-43bd-867d-106f11547d5a" (UID: "6e73e4ee-008e-43bd-867d-106f11547d5a"). InnerVolumeSpecName "kube-api-access-l5gsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:38.608835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.596856    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-kube-api-access-8rcrc" pod "a15f3912-65fe-4d9f-8b45-3da75b4c56d3" (UID: "a15f3912-65fe-4d9f-8b45-3da75b4c56d3"). InnerVolumeSpecName "kube-api-access-8rcrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:38.608835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.597136    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-xks-ca" pod "6e73e4ee-008e-43bd-867d-106f11547d5a" (UID: "6e73e4ee-008e-43bd-867d-106f11547d5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:38.659815 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.659651    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:38.659990 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.659828    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73e4ee-008e-43bd-867d-106f11547d5a-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:38.659990 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.659848    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:38.660155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.659983    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5gsh\" (UniqueName: \"kubernetes.io/projected/6e73e4ee-008e-43bd-867d-106f11547d5a-kube-api-access-l5gsh\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:38.660155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.660010    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-kube-api-access-8rcrc\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:38.690881 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.690839    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-xks-ca" pod "a15f3912-65fe-4d9f-8b45-3da75b4c56d3" (UID: "a15f3912-65fe-4d9f-8b45-3da75b4c56d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:38.761663 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.761632    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f3912-65fe-4d9f-8b45-3da75b4c56d3-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:38.995718 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.995675    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95dd232-389e-456d-accb-65be45758b49" path="/var/lib/kubelet/pods/a95dd232-389e-456d-accb-65be45758b49/volumes"
Oct 29 23:00:38.996389 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:38.996367    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8cfc30-4859-4b97-8375-aa2ec37a009f" path="/var/lib/kubelet/pods/bc8cfc30-4859-4b97-8375-aa2ec37a009f/volumes"
Oct 29 23:00:39.294402 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:39.294328    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"
Oct 29 23:00:39.294402 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:39.294359    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5" event={"ID":"6e73e4ee-008e-43bd-867d-106f11547d5a","Type":"ContainerDied","Data":"cad629d2e1c7b6f9a0a20d494acef622f4f30f86b3c540f8235ee8e6a7c0a7f0"}
Oct 29 23:00:39.294402 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:39.294401    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad629d2e1c7b6f9a0a20d494acef622f4f30f86b3c540f8235ee8e6a7c0a7f0"
Oct 29 23:00:39.295918 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:39.295897    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"
Oct 29 23:00:39.296039 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:39.295937    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49" event={"ID":"a15f3912-65fe-4d9f-8b45-3da75b4c56d3","Type":"ContainerDied","Data":"48dc8ff956c02dea960fd0877cb11e63e90e4f07c64cb0587364560627a8df71"}
Oct 29 23:00:39.296039 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:39.295979    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48dc8ff956c02dea960fd0877cb11e63e90e4f07c64cb0587364560627a8df71"
Oct 29 23:00:40.634205 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634172    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"]
Oct 29 23:00:40.634752 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634735    4742 cpu_manager.go:414] "containerMap: removing container" podUID="a15f3912-65fe-4d9f-8b45-3da75b4c56d3" containerName="revoke-keys"
Oct 29 23:00:40.634796 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634753    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15f3912-65fe-4d9f-8b45-3da75b4c56d3" containerName="revoke-keys"
Oct 29 23:00:40.634796 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634774    4742 cpu_manager.go:414] "containerMap: removing container" podUID="6e73e4ee-008e-43bd-867d-106f11547d5a" containerName="revoke-keys"
Oct 29 23:00:40.634796 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634779    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e73e4ee-008e-43bd-867d-106f11547d5a" containerName="revoke-keys"
Oct 29 23:00:40.634796 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634791    4742 cpu_manager.go:414] "containerMap: removing container" podUID="bc8cfc30-4859-4b97-8375-aa2ec37a009f" containerName="payload-pre-processing"
Oct 29 23:00:40.634923 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634797    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8cfc30-4859-4b97-8375-aa2ec37a009f" containerName="payload-pre-processing"
Oct 29 23:00:40.634923 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634812    4742 cpu_manager.go:414] "containerMap: removing container" podUID="a95dd232-389e-456d-accb-65be45758b49" containerName="payload-processing"
Oct 29 23:00:40.634923 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634818    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95dd232-389e-456d-accb-65be45758b49" containerName="payload-processing"
Oct 29 23:00:40.634923 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634899    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15f3912-65fe-4d9f-8b45-3da75b4c56d3" containerName="revoke-keys"
Oct 29 23:00:40.634923 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634910    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e73e4ee-008e-43bd-867d-106f11547d5a" containerName="revoke-keys"
Oct 29 23:00:40.634923 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634917    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8cfc30-4859-4b97-8375-aa2ec37a009f" containerName="payload-pre-processing"
Oct 29 23:00:40.634923 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.634924    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95dd232-389e-456d-accb-65be45758b49" containerName="payload-processing"
Oct 29 23:00:40.639980 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.639961    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.669140 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.669111    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"]
Oct 29 23:00:40.671367 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.671349    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:00:40.673256 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.673241    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.676663 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.676498    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:40.682869 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.682839    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"]
Oct 29 23:00:40.697304 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.696920    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"]
Oct 29 23:00:40.788783 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.788748    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf46z\" (UniqueName: \"kubernetes.io/projected/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-kube-api-access-kf46z\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.788968 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.788834    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.788968 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.788900    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.789118 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.788983    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.789118 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.789027    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxnj\" (UniqueName: \"kubernetes.io/projected/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-kube-api-access-qfxnj\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.789225 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.789147    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.890264 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.890179    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf46z\" (UniqueName: \"kubernetes.io/projected/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-kube-api-access-kf46z\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.890264 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.890250    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.890480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.890280    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.890480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.890298    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.890480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.890317    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxnj\" (UniqueName: \"kubernetes.io/projected/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-kube-api-access-qfxnj\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.890480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.890370    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.891018 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.890991    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.891018 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.891002    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.892650 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.892622    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.892769 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.892687    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.910054 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.910024    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf46z\" (UniqueName: \"kubernetes.io/projected/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-kube-api-access-kf46z\") pod \"maas-api-revoke-sub-6aa5dc33-9tnhx\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:40.910193 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.910058    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxnj\" (UniqueName: \"kubernetes.io/projected/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-kube-api-access-qfxnj\") pod \"maas-api-revoke-sub-b8115d7d-b6rbd\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:40.953352 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:40.953311    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:41.000813 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:41.000352    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:41.328029 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:41.327988    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3227003a_fe22_4483_bd7f_20a9d0cc6a5b.slice/crio-ccf16da5e4648912978834b833c1965423f6f23adfb980dcc3c18bb3129b4f70 WatchSource:0}: Error finding container ccf16da5e4648912978834b833c1965423f6f23adfb980dcc3c18bb3129b4f70: Status 404 returned error can't find the container with id ccf16da5e4648912978834b833c1965423f6f23adfb980dcc3c18bb3129b4f70
Oct 29 23:00:41.330614 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:41.330583    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"]
Oct 29 23:00:41.468524 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:41.468429    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"]
Oct 29 23:00:41.482959 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:41.482918    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-28ed1cea-c9swj"]
Oct 29 23:00:41.545494 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:41.545461    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42fa266_4ef0_4806_b8d1_dabcbcf8d126.slice/crio-57902a9c2393d493bdc1cd024252752c73b75a9fc7ed20437495576751a7b570 WatchSource:0}: Error finding container 57902a9c2393d493bdc1cd024252752c73b75a9fc7ed20437495576751a7b570: Status 404 returned error can't find the container with id 57902a9c2393d493bdc1cd024252752c73b75a9fc7ed20437495576751a7b570
Oct 29 23:00:41.545870 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:41.545848    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"]
Oct 29 23:00:41.635917 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:41.635872    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"]
Oct 29 23:00:41.636417 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:41.636143    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy" containerID="cri-o://d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5" gracePeriod=30
Oct 29 23:00:42.113405 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.113355    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" podUID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" containerName="payload-pre-processing" containerID="cri-o://b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8" gracePeriod=25
Oct 29 23:00:42.173902 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.173860    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" podUID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:42.324296 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.324260    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx" event={"ID":"e42fa266-4ef0-4806-b8d1-dabcbcf8d126","Type":"ContainerStarted","Data":"b499c4b2169e6793ed18d0789958711fcffeef4f9a52de800c7cce72496486a4"}
Oct 29 23:00:42.324446 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.324312    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx" event={"ID":"e42fa266-4ef0-4806-b8d1-dabcbcf8d126","Type":"ContainerStarted","Data":"57902a9c2393d493bdc1cd024252752c73b75a9fc7ed20437495576751a7b570"}
Oct 29 23:00:42.325932 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.325907    4742 generic.go:356] "Generic (PLEG): container finished" podID="3227003a-fe22-4483-bd7f-20a9d0cc6a5b" containerID="ee33569cd7d2153d07876c4bda581ab3d7bf9bdc398a16d7791334cdb72d21f6" exitCode=0
Oct 29 23:00:42.326029 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.325974    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd" event={"ID":"3227003a-fe22-4483-bd7f-20a9d0cc6a5b","Type":"ContainerDied","Data":"ee33569cd7d2153d07876c4bda581ab3d7bf9bdc398a16d7791334cdb72d21f6"}
Oct 29 23:00:42.326029 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.326015    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd" event={"ID":"3227003a-fe22-4483-bd7f-20a9d0cc6a5b","Type":"ContainerStarted","Data":"ccf16da5e4648912978834b833c1965423f6f23adfb980dcc3c18bb3129b4f70"}
Oct 29 23:00:42.363452 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.363354    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx" podStartSLOduration=2.363338976 podStartE2EDuration="2.363338976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:42.362407444 +0000 UTC m=+1099.503149033" watchObservedRunningTime="2026-10-29 23:00:42.363338976 +0000 UTC m=+1099.504080559"
Oct 29 23:00:42.869585 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.869547    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8" podUID="2e961f1d-5b31-4639-98a8-018c1060b806" containerName="payload-processing" containerID="cri-o://bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb" gracePeriod=25
Oct 29 23:00:42.992631 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:42.992595    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1990d261-f5fb-4ade-94f9-1f15a3f3106f" path="/var/lib/kubelet/pods/1990d261-f5fb-4ade-94f9-1f15a3f3106f/volumes"
Oct 29 23:00:43.334638 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.334607    4742 generic.go:356] "Generic (PLEG): container finished" podID="e42fa266-4ef0-4806-b8d1-dabcbcf8d126" containerID="b499c4b2169e6793ed18d0789958711fcffeef4f9a52de800c7cce72496486a4" exitCode=0
Oct 29 23:00:43.334795 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.334695    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx" event={"ID":"e42fa266-4ef0-4806-b8d1-dabcbcf8d126","Type":"ContainerDied","Data":"b499c4b2169e6793ed18d0789958711fcffeef4f9a52de800c7cce72496486a4"}
Oct 29 23:00:43.492681 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.492658    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:43.657065 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.656968    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-xks-ca\") pod \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") "
Oct 29 23:00:43.657447 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.657413    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-kube-api-access-qfxnj\" (UniqueName: \"kubernetes.io/projected/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-kube-api-access-qfxnj\") pod \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") "
Oct 29 23:00:43.657593 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.657571    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-openshift-ca\") pod \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\" (UID: \"3227003a-fe22-4483-bd7f-20a9d0cc6a5b\") "
Oct 29 23:00:43.658698 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.658656    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-openshift-ca" pod "3227003a-fe22-4483-bd7f-20a9d0cc6a5b" (UID: "3227003a-fe22-4483-bd7f-20a9d0cc6a5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:43.668268 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.668201    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-xks-ca" pod "3227003a-fe22-4483-bd7f-20a9d0cc6a5b" (UID: "3227003a-fe22-4483-bd7f-20a9d0cc6a5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:43.670828 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.670787    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-kube-api-access-qfxnj" pod "3227003a-fe22-4483-bd7f-20a9d0cc6a5b" (UID: "3227003a-fe22-4483-bd7f-20a9d0cc6a5b"). InnerVolumeSpecName "kube-api-access-qfxnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:43.759433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.759392    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:43.759433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.759425    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:43.759433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:43.759436    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxnj\" (UniqueName: \"kubernetes.io/projected/3227003a-fe22-4483-bd7f-20a9d0cc6a5b-kube-api-access-qfxnj\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.343352 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.343322    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"
Oct 29 23:00:44.343775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.343359    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd" event={"ID":"3227003a-fe22-4483-bd7f-20a9d0cc6a5b","Type":"ContainerDied","Data":"ccf16da5e4648912978834b833c1965423f6f23adfb980dcc3c18bb3129b4f70"}
Oct 29 23:00:44.343775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.343414    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf16da5e4648912978834b833c1965423f6f23adfb980dcc3c18bb3129b4f70"
Oct 29 23:00:44.413446 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.413402    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main" containerID="cri-o://922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df" gracePeriod=45
Oct 29 23:00:44.548675 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.548644    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:44.570168 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.570118    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-openshift-ca\") pod \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") "
Oct 29 23:00:44.570359 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.570176    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-kube-api-access-kf46z\" (UniqueName: \"kubernetes.io/projected/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-kube-api-access-kf46z\") pod \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") "
Oct 29 23:00:44.570632 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.570607    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-xks-ca\") pod \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\" (UID: \"e42fa266-4ef0-4806-b8d1-dabcbcf8d126\") "
Oct 29 23:00:44.570983 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.570955    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-openshift-ca" pod "e42fa266-4ef0-4806-b8d1-dabcbcf8d126" (UID: "e42fa266-4ef0-4806-b8d1-dabcbcf8d126"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:44.571276 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.571245    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.573374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.573352    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-xks-ca" pod "e42fa266-4ef0-4806-b8d1-dabcbcf8d126" (UID: "e42fa266-4ef0-4806-b8d1-dabcbcf8d126"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:44.575742 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.575715    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-kube-api-access-kf46z" pod "e42fa266-4ef0-4806-b8d1-dabcbcf8d126" (UID: "e42fa266-4ef0-4806-b8d1-dabcbcf8d126"). InnerVolumeSpecName "kube-api-access-kf46z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:44.677725 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.672902    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.677725 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.672939    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf46z\" (UniqueName: \"kubernetes.io/projected/e42fa266-4ef0-4806-b8d1-dabcbcf8d126-kube-api-access-kf46z\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.716909 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.716885    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 23:00:44.774399 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.774368    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-model-cache\") pod \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") "
Oct 29 23:00:44.774589 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.774471    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763c2875-5e58-47d8-9b2b-a9b50b8384e0-kube-api-access-7ttqm\" (UniqueName: \"kubernetes.io/projected/763c2875-5e58-47d8-9b2b-a9b50b8384e0-kube-api-access-7ttqm\") pod \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") "
Oct 29 23:00:44.774589 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.774530    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-dshm\") pod \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") "
Oct 29 23:00:44.774698 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.774619    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tls-certs\") pod \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") "
Oct 29 23:00:44.774698 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.774673    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-home\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-home\") pod \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") "
Oct 29 23:00:44.774698 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.774670    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-model-cache" pod "763c2875-5e58-47d8-9b2b-a9b50b8384e0" (UID: "763c2875-5e58-47d8-9b2b-a9b50b8384e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.774915 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.774787    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tmp-dir\") pod \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\" (UID: \"763c2875-5e58-47d8-9b2b-a9b50b8384e0\") "
Oct 29 23:00:44.776125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.775326    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.776125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.775548    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tmp-dir" pod "763c2875-5e58-47d8-9b2b-a9b50b8384e0" (UID: "763c2875-5e58-47d8-9b2b-a9b50b8384e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.776125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.775566    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-home" pod "763c2875-5e58-47d8-9b2b-a9b50b8384e0" (UID: "763c2875-5e58-47d8-9b2b-a9b50b8384e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.779139 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.777944    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tls-certs" pod "763c2875-5e58-47d8-9b2b-a9b50b8384e0" (UID: "763c2875-5e58-47d8-9b2b-a9b50b8384e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:44.779139 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.778107    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-dshm" pod "763c2875-5e58-47d8-9b2b-a9b50b8384e0" (UID: "763c2875-5e58-47d8-9b2b-a9b50b8384e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.779940 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.779875    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763c2875-5e58-47d8-9b2b-a9b50b8384e0-kube-api-access-7ttqm" pod "763c2875-5e58-47d8-9b2b-a9b50b8384e0" (UID: "763c2875-5e58-47d8-9b2b-a9b50b8384e0"). InnerVolumeSpecName "kube-api-access-7ttqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:44.876809 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.876706    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.876809 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.876739    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ttqm\" (UniqueName: \"kubernetes.io/projected/763c2875-5e58-47d8-9b2b-a9b50b8384e0-kube-api-access-7ttqm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.876809 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.876751    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.876809 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.876763    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763c2875-5e58-47d8-9b2b-a9b50b8384e0-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.876809 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:44.876774    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763c2875-5e58-47d8-9b2b-a9b50b8384e0-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:45.093993 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.093956    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"]
Oct 29 23:00:45.094526 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094511    4742 cpu_manager.go:414] "containerMap: removing container" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main"
Oct 29 23:00:45.094526 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094526    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main"
Oct 29 23:00:45.094665 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094537    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e42fa266-4ef0-4806-b8d1-dabcbcf8d126" containerName="revoke-keys"
Oct 29 23:00:45.094665 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094542    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42fa266-4ef0-4806-b8d1-dabcbcf8d126" containerName="revoke-keys"
Oct 29 23:00:45.094665 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094558    4742 cpu_manager.go:414] "containerMap: removing container" podUID="3227003a-fe22-4483-bd7f-20a9d0cc6a5b" containerName="revoke-keys"
Oct 29 23:00:45.094665 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094564    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3227003a-fe22-4483-bd7f-20a9d0cc6a5b" containerName="revoke-keys"
Oct 29 23:00:45.094665 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094665    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerName="main"
Oct 29 23:00:45.094824 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094672    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42fa266-4ef0-4806-b8d1-dabcbcf8d126" containerName="revoke-keys"
Oct 29 23:00:45.094824 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.094681    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="3227003a-fe22-4483-bd7f-20a9d0cc6a5b" containerName="revoke-keys"
Oct 29 23:00:45.098277 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.098261    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.120204 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.120170    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"]
Oct 29 23:00:45.180845 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.180737    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhdc\" (UniqueName: \"kubernetes.io/projected/d0abd2b2-5442-409c-ac11-ff9491ddb54b-kube-api-access-nfhdc\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.180993 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.180943    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.181036 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.181008    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.197893 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.197860    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm" podUID="fb308604-255e-4f3e-8c89-8b9b571e19da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:45.282667 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.282627    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.282842 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.282680    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.282924 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.282829    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhdc\" (UniqueName: \"kubernetes.io/projected/d0abd2b2-5442-409c-ac11-ff9491ddb54b-kube-api-access-nfhdc\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.283392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.283370    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.284985 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.284960    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.302397 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.302371    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhdc\" (UniqueName: \"kubernetes.io/projected/d0abd2b2-5442-409c-ac11-ff9491ddb54b-kube-api-access-nfhdc\") pod \"maas-api-revoke-sub-32ac7de2-5q777\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.352415 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.352386    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"
Oct 29 23:00:45.352801 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.352424    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx" event={"ID":"e42fa266-4ef0-4806-b8d1-dabcbcf8d126","Type":"ContainerDied","Data":"57902a9c2393d493bdc1cd024252752c73b75a9fc7ed20437495576751a7b570"}
Oct 29 23:00:45.352801 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.352479    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57902a9c2393d493bdc1cd024252752c73b75a9fc7ed20437495576751a7b570"
Oct 29 23:00:45.353852 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.353830    4742 generic.go:356] "Generic (PLEG): container finished" podID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" containerID="922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df" exitCode=0
Oct 29 23:00:45.353959 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.353887    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"
Oct 29 23:00:45.353959 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.353924    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" event={"ID":"763c2875-5e58-47d8-9b2b-a9b50b8384e0","Type":"ContainerDied","Data":"922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df"}
Oct 29 23:00:45.354043 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.353978    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr" event={"ID":"763c2875-5e58-47d8-9b2b-a9b50b8384e0","Type":"ContainerDied","Data":"1dee8501a8a799f7d7645bd152c98f2d44225503280e0e904ea96c927482af00"}
Oct 29 23:00:45.354043 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.354003    4742 scope.go:122] "RemoveContainer" containerID="922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df"
Oct 29 23:00:45.364147 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.364126    4742 scope.go:122] "RemoveContainer" containerID="922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df"
Oct 29 23:00:45.364499 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:45.364477    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df\": container with ID starting with 922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df not found: ID does not exist" containerID="922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df"
Oct 29 23:00:45.364558 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.364511    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df"} err="failed to get container status \"922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df\": rpc error: code = NotFound desc = could not find container \"922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df\": container with ID starting with 922109f7fcba763325416456c25c5466a15f11e1dfe4e10ec2f091ba9ae699df not found: ID does not exist"
Oct 29 23:00:45.411642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.411610    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:45.413875 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.413845    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"]
Oct 29 23:00:45.418806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.418782    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/premium-w2-a2e7c8-kserve-656ccb8bb4-ksrxr"]
Oct 29 23:00:45.720097 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:45.720055    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0abd2b2_5442_409c_ac11_ff9491ddb54b.slice/crio-cefb1d809dd37d036c6250f1bdd83f5d535de33c6d4144a30c171a3421dd9d8e WatchSource:0}: Error finding container cefb1d809dd37d036c6250f1bdd83f5d535de33c6d4144a30c171a3421dd9d8e: Status 404 returned error can't find the container with id cefb1d809dd37d036c6250f1bdd83f5d535de33c6d4144a30c171a3421dd9d8e
Oct 29 23:00:45.720726 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:45.720703    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"]
Oct 29 23:00:46.363587 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.363551    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777" event={"ID":"d0abd2b2-5442-409c-ac11-ff9491ddb54b","Type":"ContainerStarted","Data":"f3815852135929186087bd2fd09733ed1552d0b4042b51033e82b0a7ceb1e930"}
Oct 29 23:00:46.363975 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.363610    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777" event={"ID":"d0abd2b2-5442-409c-ac11-ff9491ddb54b","Type":"ContainerStarted","Data":"cefb1d809dd37d036c6250f1bdd83f5d535de33c6d4144a30c171a3421dd9d8e"}
Oct 29 23:00:46.451288 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.451235    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777" podStartSLOduration=1.451221679 podStartE2EDuration="1.451221679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:46.449673968 +0000 UTC m=+1103.590415552" watchObservedRunningTime="2026-10-29 23:00:46.451221679 +0000 UTC m=+1103.591963327"
Oct 29 23:00:46.472640 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.472604    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main" containerID="cri-o://12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686" gracePeriod=45
Oct 29 23:00:46.486949 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.486912    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" containerID="cri-o://90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774" gracePeriod=45
Oct 29 23:00:46.608460 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.608419    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main" containerID="cri-o://b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3" gracePeriod=45
Oct 29 23:00:46.621315 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.621243    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 29 23:00:46.797630 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.797574    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 23:00:46.803310 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.803195    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 23:00:46.902970 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.902943    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tmp-dir\") pod \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") "
Oct 29 23:00:46.903078 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903059    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-dshm\") pod \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") "
Oct 29 23:00:46.903197 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903172    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-home\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-home\") pod \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") "
Oct 29 23:00:46.903266 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903230    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-dshm\") pod \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") "
Oct 29 23:00:46.903324 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903259    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tmp-dir" pod "d0a9b7f5-7dd9-486f-9d77-c47c4f980957" (UID: "d0a9b7f5-7dd9-486f-9d77-c47c4f980957"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.903324 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903288    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-model-cache\") pod \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") "
Oct 29 23:00:46.903427 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903373    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tmp-dir\") pod \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") "
Oct 29 23:00:46.903480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903419    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-kube-api-access-hn5tt\" (UniqueName: \"kubernetes.io/projected/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-kube-api-access-hn5tt\") pod \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") "
Oct 29 23:00:46.903480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903455    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-home" pod "d0a9b7f5-7dd9-486f-9d77-c47c4f980957" (UID: "d0a9b7f5-7dd9-486f-9d77-c47c4f980957"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.903590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903527    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-model-cache\") pod \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") "
Oct 29 23:00:46.903650 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903626    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tls-certs\") pod \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") "
Oct 29 23:00:46.903708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903668    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tls-certs\" (UniqueName: \"kubernetes.io/secret/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tls-certs\") pod \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") "
Oct 29 23:00:46.904052 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903763    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-kube-api-access-fj2bt\" (UniqueName: \"kubernetes.io/projected/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-kube-api-access-fj2bt\") pod \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\" (UID: \"d0a9b7f5-7dd9-486f-9d77-c47c4f980957\") "
Oct 29 23:00:46.904052 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903911    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-model-cache" pod "02aea966-e9cc-4ee1-93b8-5f6c173b4d91" (UID: "02aea966-e9cc-4ee1-93b8-5f6c173b4d91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.904052 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.903966    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tmp-dir" pod "02aea966-e9cc-4ee1-93b8-5f6c173b4d91" (UID: "02aea966-e9cc-4ee1-93b8-5f6c173b4d91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.904052 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.904021    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-model-cache" pod "d0a9b7f5-7dd9-486f-9d77-c47c4f980957" (UID: "d0a9b7f5-7dd9-486f-9d77-c47c4f980957"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.904357 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.904160    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-home\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-home\") pod \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\" (UID: \"02aea966-e9cc-4ee1-93b8-5f6c173b4d91\") "
Oct 29 23:00:46.904639 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.904576    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-home" pod "02aea966-e9cc-4ee1-93b8-5f6c173b4d91" (UID: "02aea966-e9cc-4ee1-93b8-5f6c173b4d91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.905421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.905128    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.905421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.905164    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.905421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.905178    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.905421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.905191    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.905421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.905202    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.905421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.905213    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.911407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.911323    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tls-certs" pod "d0a9b7f5-7dd9-486f-9d77-c47c4f980957" (UID: "d0a9b7f5-7dd9-486f-9d77-c47c4f980957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:46.911407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.911332    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-kube-api-access-fj2bt" pod "d0a9b7f5-7dd9-486f-9d77-c47c4f980957" (UID: "d0a9b7f5-7dd9-486f-9d77-c47c4f980957"). InnerVolumeSpecName "kube-api-access-fj2bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.911579 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.911427    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-dshm" pod "d0a9b7f5-7dd9-486f-9d77-c47c4f980957" (UID: "d0a9b7f5-7dd9-486f-9d77-c47c4f980957"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.911801 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.911719    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-kube-api-access-hn5tt" pod "02aea966-e9cc-4ee1-93b8-5f6c173b4d91" (UID: "02aea966-e9cc-4ee1-93b8-5f6c173b4d91"). InnerVolumeSpecName "kube-api-access-hn5tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.912296 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.912263    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tls-certs" pod "02aea966-e9cc-4ee1-93b8-5f6c173b4d91" (UID: "02aea966-e9cc-4ee1-93b8-5f6c173b4d91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:46.913418 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.913395    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-dshm" pod "02aea966-e9cc-4ee1-93b8-5f6c173b4d91" (UID: "02aea966-e9cc-4ee1-93b8-5f6c173b4d91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.954681 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.954648    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 23:00:46.996285 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:46.995989    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763c2875-5e58-47d8-9b2b-a9b50b8384e0" path="/var/lib/kubelet/pods/763c2875-5e58-47d8-9b2b-a9b50b8384e0/volumes"
Oct 29 23:00:47.007375 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.006873    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-dshm\") pod \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") "
Oct 29 23:00:47.007375 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.006942    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tmp-dir\") pod \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") "
Oct 29 23:00:47.007375 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.006973    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tls-certs\") pod \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") "
Oct 29 23:00:47.011349 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.007670    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-model-cache\") pod \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") "
Oct 29 23:00:47.011349 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.007724    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-home\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-home\") pod \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") "
Oct 29 23:00:47.011349 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.007782    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-kube-api-access-gb9sz\" (UniqueName: \"kubernetes.io/projected/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-kube-api-access-gb9sz\") pod \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\" (UID: \"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f\") "
Oct 29 23:00:47.011349 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.010776    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-dshm" pod "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" (UID: "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.013145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.011411    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-model-cache" pod "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" (UID: "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.013145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.011843    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-kube-api-access-gb9sz" pod "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" (UID: "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f"). InnerVolumeSpecName "kube-api-access-gb9sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:47.013145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.012041    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-home" pod "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" (UID: "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.013145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.012249    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tls-certs" pod "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" (UID: "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:47.013145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.012271    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tmp-dir" pod "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" (UID: "8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.014153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014137    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5tt\" (UniqueName: \"kubernetes.io/projected/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-kube-api-access-hn5tt\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014160    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014178    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014190    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014203    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj2bt\" (UniqueName: \"kubernetes.io/projected/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-kube-api-access-fj2bt\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014217    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014230    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014241    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02aea966-e9cc-4ee1-93b8-5f6c173b4d91-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014472 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014255    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9sz\" (UniqueName: \"kubernetes.io/projected/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-kube-api-access-gb9sz\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014472 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014269    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a9b7f5-7dd9-486f-9d77-c47c4f980957-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.014472 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.014282    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.018579 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.018551    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 23:00:47.115606 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.115572    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-credential-socket\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.115805 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.115635    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-token\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-token\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.115805 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.115694    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istiod-ca-cert\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.115805 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.115729    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-data\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.119692 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.116412    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-data" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.119692 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.116457    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istiod-ca-cert" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:47.119692 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.116510    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-credential-socket" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.125297    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-podinfo\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.125433    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-kube-api-access-h6j4t\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-kube-api-access-h6j4t\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.125525    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-socket\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.125595    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-envoy\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.125674    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-certs\") pod \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\" (UID: \"67aa34c6-25e3-4dc4-a57e-1dd90de4444c\") "
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.126547    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.126581    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.126597    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.126818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.126612    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.127640 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.127442    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-certs" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.128058 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.127781    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-socket" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.128990 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.128964    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-token" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:47.131041 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.130560    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-envoy" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:47.131653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.131632    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-podinfo" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:47.131741 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.131719    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-kube-api-access-h6j4t" pod "67aa34c6-25e3-4dc4-a57e-1dd90de4444c" (UID: "67aa34c6-25e3-4dc4-a57e-1dd90de4444c"). InnerVolumeSpecName "kube-api-access-h6j4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:47.198884 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.198808    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"]
Oct 29 23:00:47.209399 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.209362    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"]
Oct 29 23:00:47.228225 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.228192    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.228225 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.228225    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.228460 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.228240    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.228460 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.228253    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.228460 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.228268    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6j4t\" (UniqueName: \"kubernetes.io/projected/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-kube-api-access-h6j4t\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.228460 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.228285    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa34c6-25e3-4dc4-a57e-1dd90de4444c-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.237950 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.237917    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"]
Oct 29 23:00:47.238231 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.238204    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk" podUID="0cb920ed-428a-4ddd-8aac-ac2d7f931463" containerName="maas-api" containerID="cri-o://184b3c90a9ef16d8c05c1327c4b7ba767781e8744c629f59246d3f900efad5cf" gracePeriod=30
Oct 29 23:00:47.248496 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.248469    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"]
Oct 29 23:00:47.280883 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.280835    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk" podUID="0cb920ed-428a-4ddd-8aac-ac2d7f931463" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": dial tcp 10.134.0.44:8443: connect: connection refused"
Oct 29 23:00:47.372370 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.372333    4742 generic.go:356] "Generic (PLEG): container finished" podID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerID="12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686" exitCode=0
Oct 29 23:00:47.372782 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.372424    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"
Oct 29 23:00:47.372782 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.372429    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" event={"ID":"02aea966-e9cc-4ee1-93b8-5f6c173b4d91","Type":"ContainerDied","Data":"12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686"}
Oct 29 23:00:47.372782 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.372491    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng" event={"ID":"02aea966-e9cc-4ee1-93b8-5f6c173b4d91","Type":"ContainerDied","Data":"08e0086ad617b072125f966456c506a8895b52bc53dbe10952811cd613f7f636"}
Oct 29 23:00:47.372782 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.372510    4742 scope.go:122] "RemoveContainer" containerID="12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686"
Oct 29 23:00:47.374055 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.374021    4742 generic.go:356] "Generic (PLEG): container finished" podID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerID="b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3" exitCode=0
Oct 29 23:00:47.374164 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.374101    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"
Oct 29 23:00:47.374240 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.374212    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" event={"ID":"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f","Type":"ContainerDied","Data":"b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3"}
Oct 29 23:00:47.374292 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.374247    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb" event={"ID":"8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f","Type":"ContainerDied","Data":"81d39d48647b6a1beae187fb7908ce6ab8d85ed3200e3a00b502195cf9f38cca"}
Oct 29 23:00:47.375916 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.375893    4742 generic.go:356] "Generic (PLEG): container finished" podID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerID="90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774" exitCode=0
Oct 29 23:00:47.376016 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.376002    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" event={"ID":"d0a9b7f5-7dd9-486f-9d77-c47c4f980957","Type":"ContainerDied","Data":"90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774"}
Oct 29 23:00:47.376069 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.376029    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk" event={"ID":"d0a9b7f5-7dd9-486f-9d77-c47c4f980957","Type":"ContainerDied","Data":"61f8d5be1f09cd3c14f06504d736e6a8df48b8f5abed68ec7340a61e2ed80937"}
Oct 29 23:00:47.376158 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.376107    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"
Oct 29 23:00:47.378369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.378350    4742 generic.go:356] "Generic (PLEG): container finished" podID="d0abd2b2-5442-409c-ac11-ff9491ddb54b" containerID="f3815852135929186087bd2fd09733ed1552d0b4042b51033e82b0a7ceb1e930" exitCode=0
Oct 29 23:00:47.378459 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.378444    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777" event={"ID":"d0abd2b2-5442-409c-ac11-ff9491ddb54b","Type":"ContainerDied","Data":"f3815852135929186087bd2fd09733ed1552d0b4042b51033e82b0a7ceb1e930"}
Oct 29 23:00:47.380444 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.380424    4742 generic.go:356] "Generic (PLEG): container finished" podID="0cb920ed-428a-4ddd-8aac-ac2d7f931463" containerID="184b3c90a9ef16d8c05c1327c4b7ba767781e8744c629f59246d3f900efad5cf" exitCode=0
Oct 29 23:00:47.380544 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.380494    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk" event={"ID":"0cb920ed-428a-4ddd-8aac-ac2d7f931463","Type":"ContainerDied","Data":"184b3c90a9ef16d8c05c1327c4b7ba767781e8744c629f59246d3f900efad5cf"}
Oct 29 23:00:47.382201 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.382184    4742 generic.go:356] "Generic (PLEG): container finished" podID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerID="d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5" exitCode=0
Oct 29 23:00:47.382320 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.382217    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" event={"ID":"67aa34c6-25e3-4dc4-a57e-1dd90de4444c","Type":"ContainerDied","Data":"d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5"}
Oct 29 23:00:47.382440 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.382340    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c" event={"ID":"67aa34c6-25e3-4dc4-a57e-1dd90de4444c","Type":"ContainerDied","Data":"6e12c15634b4ba9df897b6d3ab416fc8cfd80e13d1f6a14815174a11a8da5c26"}
Oct 29 23:00:47.382440 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.382252    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"
Oct 29 23:00:47.385926 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.385906    4742 scope.go:122] "RemoveContainer" containerID="12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686"
Oct 29 23:00:47.386261 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:47.386241    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686\": container with ID starting with 12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686 not found: ID does not exist" containerID="12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686"
Oct 29 23:00:47.386344 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.386275    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686"} err="failed to get container status \"12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686\": rpc error: code = NotFound desc = could not find container \"12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686\": container with ID starting with 12a63b4103592f9ac166ae6257cd0cec829c893d4688637dfaca88051fa67686 not found: ID does not exist"
Oct 29 23:00:47.386344 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.386298    4742 scope.go:122] "RemoveContainer" containerID="b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3"
Oct 29 23:00:47.396747 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.396725    4742 scope.go:122] "RemoveContainer" containerID="b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3"
Oct 29 23:00:47.397047 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:47.397029    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3\": container with ID starting with b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3 not found: ID does not exist" containerID="b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3"
Oct 29 23:00:47.397206 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.397052    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3"} err="failed to get container status \"b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3\": rpc error: code = NotFound desc = could not find container \"b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3\": container with ID starting with b3dde8fd10cf25894d99f91d19689c678969d7e8f37071df678ba3c34f0b53f3 not found: ID does not exist"
Oct 29 23:00:47.397206 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.397068    4742 scope.go:122] "RemoveContainer" containerID="90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774"
Oct 29 23:00:47.406853 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.406822    4742 scope.go:122] "RemoveContainer" containerID="90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774"
Oct 29 23:00:47.407163 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:47.407139    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774\": container with ID starting with 90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774 not found: ID does not exist" containerID="90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774"
Oct 29 23:00:47.407275 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.407169    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774"} err="failed to get container status \"90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774\": rpc error: code = NotFound desc = could not find container \"90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774\": container with ID starting with 90a697d514c3234de214a7b3ca01586b51fc5dd097925cc7093750cb64c57774 not found: ID does not exist"
Oct 29 23:00:47.407275 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.407187    4742 scope.go:122] "RemoveContainer" containerID="d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5"
Oct 29 23:00:47.418169 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.418143    4742 scope.go:122] "RemoveContainer" containerID="4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657"
Oct 29 23:00:47.432829 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.432798    4742 scope.go:122] "RemoveContainer" containerID="d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5"
Oct 29 23:00:47.433247 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:47.433222    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5\": container with ID starting with d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5 not found: ID does not exist" containerID="d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5"
Oct 29 23:00:47.433343 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.433260    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5"} err="failed to get container status \"d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5\": rpc error: code = NotFound desc = could not find container \"d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5\": container with ID starting with d49c68c494cdf6b276ce2ba96c4ab2b3d8457d1b3664e8d4ad101ef014ab23f5 not found: ID does not exist"
Oct 29 23:00:47.433343 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.433286    4742 scope.go:122] "RemoveContainer" containerID="4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657"
Oct 29 23:00:47.433615 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:47.433594    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657\": container with ID starting with 4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657 not found: ID does not exist" containerID="4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657"
Oct 29 23:00:47.433669 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.433624    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657"} err="failed to get container status \"4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657\": rpc error: code = NotFound desc = could not find container \"4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657\": container with ID starting with 4dc85ca38c9c4d587a6855e12498d82f5b227662c4b8d9e0171026b320223657 not found: ID does not exist"
Oct 29 23:00:47.528181 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.528153    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"]
Oct 29 23:00:47.532293 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.532269    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/premium-w6-81f92d-kserve-96c68c86d-9bklk"]
Oct 29 23:00:47.581542 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.581438    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 23:00:47.587943 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.587904    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"]
Oct 29 23:00:47.606141 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.600826    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-28ed1cea-openshift-default-867d9f9597-lzj8c"]
Oct 29 23:00:47.631869 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.631837    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"]
Oct 29 23:00:47.633045 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.633011    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-metrics-certs\") pod \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") "
Oct 29 23:00:47.633233 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.633195    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb920ed-428a-4ddd-8aac-ac2d7f931463-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb920ed-428a-4ddd-8aac-ac2d7f931463-openshift-service-ca\") pod \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") "
Oct 29 23:00:47.633439 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.633416    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb920ed-428a-4ddd-8aac-ac2d7f931463-kube-api-access-rwxq4\" (UniqueName: \"kubernetes.io/projected/0cb920ed-428a-4ddd-8aac-ac2d7f931463-kube-api-access-rwxq4\") pod \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") "
Oct 29 23:00:47.633518 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.633497    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-maas-api-tls\") pod \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\" (UID: \"0cb920ed-428a-4ddd-8aac-ac2d7f931463\") "
Oct 29 23:00:47.634809 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.634777    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb920ed-428a-4ddd-8aac-ac2d7f931463-openshift-service-ca" pod "0cb920ed-428a-4ddd-8aac-ac2d7f931463" (UID: "0cb920ed-428a-4ddd-8aac-ac2d7f931463"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:47.636593 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.636566    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-maas-api-tls" pod "0cb920ed-428a-4ddd-8aac-ac2d7f931463" (UID: "0cb920ed-428a-4ddd-8aac-ac2d7f931463"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:47.636706 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.636651    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb920ed-428a-4ddd-8aac-ac2d7f931463-kube-api-access-rwxq4" pod "0cb920ed-428a-4ddd-8aac-ac2d7f931463" (UID: "0cb920ed-428a-4ddd-8aac-ac2d7f931463"). InnerVolumeSpecName "kube-api-access-rwxq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:47.636770 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.636713    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-metrics-certs" pod "0cb920ed-428a-4ddd-8aac-ac2d7f931463" (UID: "0cb920ed-428a-4ddd-8aac-ac2d7f931463"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:47.639387 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.639364    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-w6-81f92d-kserve-7ccb58489f-lzgng"]
Oct 29 23:00:47.723476 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.723440    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"]
Oct 29 23:00:47.735356 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.735295    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/simulator-w1-20f88f-kserve-6dbff847f8-zvjvb"]
Oct 29 23:00:47.735822 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.735785    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwxq4\" (UniqueName: \"kubernetes.io/projected/0cb920ed-428a-4ddd-8aac-ac2d7f931463-kube-api-access-rwxq4\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.735872 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.735824    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-maas-api-tls\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.735872 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.735838    4742 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb920ed-428a-4ddd-8aac-ac2d7f931463-metrics-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.735872 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:47.735851    4742 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb920ed-428a-4ddd-8aac-ac2d7f931463-openshift-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:48.398941 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.398901    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk" event={"ID":"0cb920ed-428a-4ddd-8aac-ac2d7f931463","Type":"ContainerDied","Data":"a2397ea0721da8b6ed45bc621164e455ca8287b35e14e4cda216ae3736b5319b"}
Oct 29 23:00:48.399631 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.399606    4742 scope.go:122] "RemoveContainer" containerID="184b3c90a9ef16d8c05c1327c4b7ba767781e8744c629f59246d3f900efad5cf"
Oct 29 23:00:48.399749 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.398951    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"
Oct 29 23:00:48.462099 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.462057    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"]
Oct 29 23:00:48.477677 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.477645    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bf55a-78648b6685-vhmgk"]
Oct 29 23:00:48.572747 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.572718    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:48.701124 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.701000    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-xks-ca\") pod \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") "
Oct 29 23:00:48.701124 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.701072    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0abd2b2-5442-409c-ac11-ff9491ddb54b-kube-api-access-nfhdc\" (UniqueName: \"kubernetes.io/projected/d0abd2b2-5442-409c-ac11-ff9491ddb54b-kube-api-access-nfhdc\") pod \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") "
Oct 29 23:00:48.701349 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.701135    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-openshift-ca\") pod \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\" (UID: \"d0abd2b2-5442-409c-ac11-ff9491ddb54b\") "
Oct 29 23:00:48.701485 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.701461    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-openshift-ca" pod "d0abd2b2-5442-409c-ac11-ff9491ddb54b" (UID: "d0abd2b2-5442-409c-ac11-ff9491ddb54b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:48.701638 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.701623    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:48.703159 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.703134    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-xks-ca" pod "d0abd2b2-5442-409c-ac11-ff9491ddb54b" (UID: "d0abd2b2-5442-409c-ac11-ff9491ddb54b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:48.703242 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.703175    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0abd2b2-5442-409c-ac11-ff9491ddb54b-kube-api-access-nfhdc" pod "d0abd2b2-5442-409c-ac11-ff9491ddb54b" (UID: "d0abd2b2-5442-409c-ac11-ff9491ddb54b"). InnerVolumeSpecName "kube-api-access-nfhdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:48.803338 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.803304    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0abd2b2-5442-409c-ac11-ff9491ddb54b-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:48.803338 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.803332    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfhdc\" (UniqueName: \"kubernetes.io/projected/d0abd2b2-5442-409c-ac11-ff9491ddb54b-kube-api-access-nfhdc\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:48.991736 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.991653    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" path="/var/lib/kubelet/pods/02aea966-e9cc-4ee1-93b8-5f6c173b4d91/volumes"
Oct 29 23:00:48.992249 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.992223    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb920ed-428a-4ddd-8aac-ac2d7f931463" path="/var/lib/kubelet/pods/0cb920ed-428a-4ddd-8aac-ac2d7f931463/volumes"
Oct 29 23:00:48.992846 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.992823    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" path="/var/lib/kubelet/pods/67aa34c6-25e3-4dc4-a57e-1dd90de4444c/volumes"
Oct 29 23:00:48.993454 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.993435    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" path="/var/lib/kubelet/pods/8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f/volumes"
Oct 29 23:00:48.993844 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:48.993829    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" path="/var/lib/kubelet/pods/d0a9b7f5-7dd9-486f-9d77-c47c4f980957/volumes"
Oct 29 23:00:49.225391 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:49.225354    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8" podUID="2e961f1d-5b31-4639-98a8-018c1060b806" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:49.417852 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:49.417823    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777" event={"ID":"d0abd2b2-5442-409c-ac11-ff9491ddb54b","Type":"ContainerDied","Data":"cefb1d809dd37d036c6250f1bdd83f5d535de33c6d4144a30c171a3421dd9d8e"}
Oct 29 23:00:49.418253 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:49.417870    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cefb1d809dd37d036c6250f1bdd83f5d535de33c6d4144a30c171a3421dd9d8e"
Oct 29 23:00:49.418253 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:49.417871    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"
Oct 29 23:00:52.174783 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:52.174738    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" podUID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:52.771270 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:52.771235    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"]
Oct 29 23:00:52.802110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:52.802052    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"]
Oct 29 23:00:53.130183 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:53.130136    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:00:54.288627 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.288603    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 23:00:54.335307 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.335274    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb308604-255e-4f3e-8c89-8b9b571e19da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb308604-255e-4f3e-8c89-8b9b571e19da-plugins-config-volume\") pod \"fb308604-255e-4f3e-8c89-8b9b571e19da\" (UID: \"fb308604-255e-4f3e-8c89-8b9b571e19da\") "
Oct 29 23:00:54.335519 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.335501    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb308604-255e-4f3e-8c89-8b9b571e19da-kube-api-access-clbns\" (UniqueName: \"kubernetes.io/projected/fb308604-255e-4f3e-8c89-8b9b571e19da-kube-api-access-clbns\") pod \"fb308604-255e-4f3e-8c89-8b9b571e19da\" (UID: \"fb308604-255e-4f3e-8c89-8b9b571e19da\") "
Oct 29 23:00:54.335728 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.335705    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb308604-255e-4f3e-8c89-8b9b571e19da-plugins-config-volume" pod "fb308604-255e-4f3e-8c89-8b9b571e19da" (UID: "fb308604-255e-4f3e-8c89-8b9b571e19da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:54.336006 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.335987    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb308604-255e-4f3e-8c89-8b9b571e19da-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:54.337516 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.337492    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb308604-255e-4f3e-8c89-8b9b571e19da-kube-api-access-clbns" pod "fb308604-255e-4f3e-8c89-8b9b571e19da" (UID: "fb308604-255e-4f3e-8c89-8b9b571e19da"). InnerVolumeSpecName "kube-api-access-clbns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:54.437720 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.437665    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clbns\" (UniqueName: \"kubernetes.io/projected/fb308604-255e-4f3e-8c89-8b9b571e19da-kube-api-access-clbns\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:54.450902 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.450866    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"]
Oct 29 23:00:54.465528 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.465497    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-81f92d-mc7dl"]
Oct 29 23:00:54.508235 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.508198    4742 generic.go:356] "Generic (PLEG): container finished" podID="fb308604-255e-4f3e-8c89-8b9b571e19da" containerID="f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf" exitCode=0
Oct 29 23:00:54.508406 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.508248    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"
Oct 29 23:00:54.508406 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.508278    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm" event={"ID":"fb308604-255e-4f3e-8c89-8b9b571e19da","Type":"ContainerDied","Data":"f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf"}
Oct 29 23:00:54.508406 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.508342    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm" event={"ID":"fb308604-255e-4f3e-8c89-8b9b571e19da","Type":"ContainerDied","Data":"4bba2981cdf31438b800924d3e13952d52dab0285cc3f9e7337d574c2d4d106a"}
Oct 29 23:00:54.508406 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.508370    4742 scope.go:122] "RemoveContainer" containerID="f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf"
Oct 29 23:00:54.522463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.522440    4742 scope.go:122] "RemoveContainer" containerID="f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf"
Oct 29 23:00:54.522769 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:00:54.522748    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf\": container with ID starting with f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf not found: ID does not exist" containerID="f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf"
Oct 29 23:00:54.522845 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.522780    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf"} err="failed to get container status \"f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf\": rpc error: code = NotFound desc = could not find container \"f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf\": container with ID starting with f57f03c892b42558c640efc006e16b4b4240d354c53614be9744d28bba1b9ccf not found: ID does not exist"
Oct 29 23:00:54.632014 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.631963    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"]
Oct 29 23:00:54.638287 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.638252    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bf55a-75d5fb77d7-lmqlm"]
Oct 29 23:00:54.816197 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.809797    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"]
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818333    4742 cpu_manager.go:414] "containerMap: removing container" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818367    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818473    4742 cpu_manager.go:414] "containerMap: removing container" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818482    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818507    4742 cpu_manager.go:414] "containerMap: removing container" podUID="fb308604-255e-4f3e-8c89-8b9b571e19da" containerName="payload-processing"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818515    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb308604-255e-4f3e-8c89-8b9b571e19da" containerName="payload-processing"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818544    4742 cpu_manager.go:414] "containerMap: removing container" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818552    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818590    4742 cpu_manager.go:414] "containerMap: removing container" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818597    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818608    4742 cpu_manager.go:414] "containerMap: removing container" podUID="0cb920ed-428a-4ddd-8aac-ac2d7f931463" containerName="maas-api"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.818614    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb920ed-428a-4ddd-8aac-ac2d7f931463" containerName="maas-api"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819199    4742 cpu_manager.go:414] "containerMap: removing container" podUID="d0abd2b2-5442-409c-ac11-ff9491ddb54b" containerName="revoke-keys"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819216    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0abd2b2-5442-409c-ac11-ff9491ddb54b" containerName="revoke-keys"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819408    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0abd2b2-5442-409c-ac11-ff9491ddb54b" containerName="revoke-keys"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819424    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbee0a3-5948-4aea-a1bc-b07f6c2cd54f" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819433    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819443    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb308604-255e-4f3e-8c89-8b9b571e19da" containerName="payload-processing"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819457    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="02aea966-e9cc-4ee1-93b8-5f6c173b4d91" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819472    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a9b7f5-7dd9-486f-9d77-c47c4f980957" containerName="main"
Oct 29 23:00:54.824107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.819482    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb920ed-428a-4ddd-8aac-ac2d7f931463" containerName="maas-api"
Oct 29 23:00:54.831752 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.831720    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:54.847005 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.846973    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"]
Oct 29 23:00:54.847876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.847850    4742 cpu_manager.go:414] "containerMap: removing container" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy"
Oct 29 23:00:54.847876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.847877    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy"
Oct 29 23:00:54.848052 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.848040    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="67aa34c6-25e3-4dc4-a57e-1dd90de4444c" containerName="istio-proxy"
Oct 29 23:00:54.852018 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.851995    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:54.859493 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.859460    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"]
Oct 29 23:00:54.868176 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.867468    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:00:54.868176 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.867629    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:00:54.868176 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.867683    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:54.868176 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.867726    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:00:54.925739 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.925708    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"]
Oct 29 23:00:54.945542 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.945506    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:54.945687 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.945578    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:54.945849 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.945691    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rxj\" (UniqueName: \"kubernetes.io/projected/42786f94-d2b2-493c-b722-2c0ef835e650-kube-api-access-n2rxj\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:54.945849 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.945733    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9cf\" (UniqueName: \"kubernetes.io/projected/a08b4d2b-445e-4ae7-a976-419fc19ec87a-kube-api-access-mh9cf\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:54.945849 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.945837    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:54.945989 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.945972    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:54.991056 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.991017    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85c4b25-31c7-4246-b7d7-3fecbff10238" path="/var/lib/kubelet/pods/d85c4b25-31c7-4246-b7d7-3fecbff10238/volumes"
Oct 29 23:00:54.991476 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:54.991462    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb308604-255e-4f3e-8c89-8b9b571e19da" path="/var/lib/kubelet/pods/fb308604-255e-4f3e-8c89-8b9b571e19da/volumes"
Oct 29 23:00:55.047482 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.047447    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:55.047654 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.047540    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:55.047654 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.047603    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:55.047654 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.047639    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rxj\" (UniqueName: \"kubernetes.io/projected/42786f94-d2b2-493c-b722-2c0ef835e650-kube-api-access-n2rxj\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:55.047851 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.047667    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9cf\" (UniqueName: \"kubernetes.io/projected/a08b4d2b-445e-4ae7-a976-419fc19ec87a-kube-api-access-mh9cf\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:55.047851 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.047720    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:55.048410 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.048380    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:55.048551 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.048533    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:55.049974 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.049946    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:55.050077 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.050019    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:55.075560 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.075490    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rxj\" (UniqueName: \"kubernetes.io/projected/42786f94-d2b2-493c-b722-2c0ef835e650-kube-api-access-n2rxj\") pod \"maas-api-revoke-sub-7de552c8-2mvv7\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:55.075560 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.075519    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9cf\" (UniqueName: \"kubernetes.io/projected/a08b4d2b-445e-4ae7-a976-419fc19ec87a-kube-api-access-mh9cf\") pod \"maas-api-revoke-sub-dda17b4e-tfqc2\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:55.159855 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.159814    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:55.173849 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.173811    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:55.196759 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.196702    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"]
Oct 29 23:00:55.201507 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.201479    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a2e7c8-vpdpf"]
Oct 29 23:00:55.486019 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.485985    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"]
Oct 29 23:00:55.505484 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.505460    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"]
Oct 29 23:00:55.516924 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:00:55.516896    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08b4d2b_445e_4ae7_a976_419fc19ec87a.slice/crio-5eda90fffe754d948effa69bba0c0f4f08aa020c03d28c57cc39e513d158a7f4 WatchSource:0}: Error finding container 5eda90fffe754d948effa69bba0c0f4f08aa020c03d28c57cc39e513d158a7f4: Status 404 returned error can't find the container with id 5eda90fffe754d948effa69bba0c0f4f08aa020c03d28c57cc39e513d158a7f4
Oct 29 23:00:55.518317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:55.518290    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7" event={"ID":"42786f94-d2b2-493c-b722-2c0ef835e650","Type":"ContainerStarted","Data":"6db25d8928129afe6e7506219805229ff22b1df4ae119844246fe8a5f7dbd3f5"}
Oct 29 23:00:56.529502 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:56.529469    4742 generic.go:356] "Generic (PLEG): container finished" podID="42786f94-d2b2-493c-b722-2c0ef835e650" containerID="1133bf8f05d5205c1106d5182f71715cbb16e54afdc06b105895bff503faceb8" exitCode=0
Oct 29 23:00:56.529933 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:56.529569    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7" event={"ID":"42786f94-d2b2-493c-b722-2c0ef835e650","Type":"ContainerDied","Data":"1133bf8f05d5205c1106d5182f71715cbb16e54afdc06b105895bff503faceb8"}
Oct 29 23:00:56.531437 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:56.531410    4742 generic.go:356] "Generic (PLEG): container finished" podID="a08b4d2b-445e-4ae7-a976-419fc19ec87a" containerID="8534d27dac7c457360cdb347fa4a67ade08a37cd93c48c5af708e62e52682cfb" exitCode=0
Oct 29 23:00:56.531545 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:56.531444    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2" event={"ID":"a08b4d2b-445e-4ae7-a976-419fc19ec87a","Type":"ContainerDied","Data":"8534d27dac7c457360cdb347fa4a67ade08a37cd93c48c5af708e62e52682cfb"}
Oct 29 23:00:56.531545 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:56.531476    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2" event={"ID":"a08b4d2b-445e-4ae7-a976-419fc19ec87a","Type":"ContainerStarted","Data":"5eda90fffe754d948effa69bba0c0f4f08aa020c03d28c57cc39e513d158a7f4"}
Oct 29 23:00:57.036515 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:57.036475    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd2e0d2-03d4-428c-af26-144dab9d04ca" path="/var/lib/kubelet/pods/3fd2e0d2-03d4-428c-af26-144dab9d04ca/volumes"
Oct 29 23:00:57.817108 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:57.805594    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4" podUID="f952eacf-1713-4dba-890b-ac9838b1636b" containerName="payload-pre-processing" containerID="cri-o://7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24" gracePeriod=25
Oct 29 23:00:57.817108 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:57.807203    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr" podUID="33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" containerName="payload-processing" containerID="cri-o://a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa" gracePeriod=25
Oct 29 23:00:58.157038 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:58.156926    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"]
Oct 29 23:00:58.163501 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:58.163165    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy" containerID="cri-o://610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5" gracePeriod=30
Oct 29 23:00:58.891000 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:58.833189    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 29 23:00:59.104371 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.103540    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:59.565108 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.561320    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:59.751929 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.751879    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8" podUID="2e961f1d-5b31-4639-98a8-018c1060b806" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:59.761059 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.761027    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:00:59.850217 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.850143    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:00:59.975797 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.975765    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-xks-ca\") pod \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") "
Oct 29 23:00:59.976306 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.975832    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42786f94-d2b2-493c-b722-2c0ef835e650-kube-api-access-n2rxj\" (UniqueName: \"kubernetes.io/projected/42786f94-d2b2-493c-b722-2c0ef835e650-kube-api-access-n2rxj\") pod \"42786f94-d2b2-493c-b722-2c0ef835e650\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") "
Oct 29 23:00:59.976306 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.975889    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08b4d2b-445e-4ae7-a976-419fc19ec87a-kube-api-access-mh9cf\" (UniqueName: \"kubernetes.io/projected/a08b4d2b-445e-4ae7-a976-419fc19ec87a-kube-api-access-mh9cf\") pod \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") "
Oct 29 23:00:59.976306 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.975938    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-openshift-ca\") pod \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\" (UID: \"a08b4d2b-445e-4ae7-a976-419fc19ec87a\") "
Oct 29 23:00:59.976306 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.975975    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-openshift-ca\") pod \"42786f94-d2b2-493c-b722-2c0ef835e650\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") "
Oct 29 23:00:59.976306 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.976034    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-xks-ca\") pod \"42786f94-d2b2-493c-b722-2c0ef835e650\" (UID: \"42786f94-d2b2-493c-b722-2c0ef835e650\") "
Oct 29 23:00:59.976545 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.976479    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-openshift-ca" pod "a08b4d2b-445e-4ae7-a976-419fc19ec87a" (UID: "a08b4d2b-445e-4ae7-a976-419fc19ec87a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:59.976868 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.976846    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.976913 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.976876    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-openshift-ca" pod "42786f94-d2b2-493c-b722-2c0ef835e650" (UID: "42786f94-d2b2-493c-b722-2c0ef835e650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:59.978556 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.978524    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42786f94-d2b2-493c-b722-2c0ef835e650-kube-api-access-n2rxj" pod "42786f94-d2b2-493c-b722-2c0ef835e650" (UID: "42786f94-d2b2-493c-b722-2c0ef835e650"). InnerVolumeSpecName "kube-api-access-n2rxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:59.980022 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.979991    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-xks-ca" pod "42786f94-d2b2-493c-b722-2c0ef835e650" (UID: "42786f94-d2b2-493c-b722-2c0ef835e650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:59.981264 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.981235    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08b4d2b-445e-4ae7-a976-419fc19ec87a-kube-api-access-mh9cf" pod "a08b4d2b-445e-4ae7-a976-419fc19ec87a" (UID: "a08b4d2b-445e-4ae7-a976-419fc19ec87a"). InnerVolumeSpecName "kube-api-access-mh9cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:59.981488 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:00:59.981433    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-xks-ca" pod "a08b4d2b-445e-4ae7-a976-419fc19ec87a" (UID: "a08b4d2b-445e-4ae7-a976-419fc19ec87a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:00.077624 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.077585    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08b4d2b-445e-4ae7-a976-419fc19ec87a-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:00.077624 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.077617    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2rxj\" (UniqueName: \"kubernetes.io/projected/42786f94-d2b2-493c-b722-2c0ef835e650-kube-api-access-n2rxj\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:00.077624 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.077628    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh9cf\" (UniqueName: \"kubernetes.io/projected/a08b4d2b-445e-4ae7-a976-419fc19ec87a-kube-api-access-mh9cf\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:00.077855 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.077639    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:00.077855 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.077648    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42786f94-d2b2-493c-b722-2c0ef835e650-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:00.177496 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.177071    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2" event={"ID":"a08b4d2b-445e-4ae7-a976-419fc19ec87a","Type":"ContainerDied","Data":"5eda90fffe754d948effa69bba0c0f4f08aa020c03d28c57cc39e513d158a7f4"}
Oct 29 23:01:00.177496 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.177077    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"
Oct 29 23:01:00.177496 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.177141    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eda90fffe754d948effa69bba0c0f4f08aa020c03d28c57cc39e513d158a7f4"
Oct 29 23:01:00.179216 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.179194    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"
Oct 29 23:01:00.179351 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.179195    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7" event={"ID":"42786f94-d2b2-493c-b722-2c0ef835e650","Type":"ContainerDied","Data":"6db25d8928129afe6e7506219805229ff22b1df4ae119844246fe8a5f7dbd3f5"}
Oct 29 23:01:00.179351 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:00.179288    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db25d8928129afe6e7506219805229ff22b1df4ae119844246fe8a5f7dbd3f5"
Oct 29 23:01:02.174860 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:01:02.174832    4742 logging.go:55] [core] [Channel #1211 SubChannel #1212]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.37:50052", ServerName: "10.134.0.37:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.37:50052: connect: connection refused"
Oct 29 23:01:02.232588 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.232550    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main" containerID="cri-o://7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6" gracePeriod=45
Oct 29 23:01:02.236751 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.236704    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main" containerID="cri-o://24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed" gracePeriod=45
Oct 29 23:01:02.273209 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.273172    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main" containerID="cri-o://977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9" gracePeriod=45
Oct 29 23:01:02.416132 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.413206    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 23:01:02.501575 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.501527    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61d8c13-0e9f-4f30-be94-cddeb5abb510-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b61d8c13-0e9f-4f30-be94-cddeb5abb510-plugins-config-volume\") pod \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\" (UID: \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\") "
Oct 29 23:01:02.501834 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.501817    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61d8c13-0e9f-4f30-be94-cddeb5abb510-kube-api-access-x6chm\" (UniqueName: \"kubernetes.io/projected/b61d8c13-0e9f-4f30-be94-cddeb5abb510-kube-api-access-x6chm\") pod \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\" (UID: \"b61d8c13-0e9f-4f30-be94-cddeb5abb510\") "
Oct 29 23:01:02.503492 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.503459    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61d8c13-0e9f-4f30-be94-cddeb5abb510-plugins-config-volume" pod "b61d8c13-0e9f-4f30-be94-cddeb5abb510" (UID: "b61d8c13-0e9f-4f30-be94-cddeb5abb510"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:02.505843 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.505806    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61d8c13-0e9f-4f30-be94-cddeb5abb510-kube-api-access-x6chm" pod "b61d8c13-0e9f-4f30-be94-cddeb5abb510" (UID: "b61d8c13-0e9f-4f30-be94-cddeb5abb510"). InnerVolumeSpecName "kube-api-access-x6chm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:02.544837 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.544798    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 23:01:02.563166 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.563134    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 23:01:02.576182 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.576149    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.602948    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tmp-dir\") pod \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603015    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf42312-d61e-4b09-8a1b-37acd752ce5c-kube-api-access-rhhzg\" (UniqueName: \"kubernetes.io/projected/1bf42312-d61e-4b09-8a1b-37acd752ce5c-kube-api-access-rhhzg\") pod \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603122    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-model-cache\") pod \"dddcb392-8e28-4cce-984c-1f06108f927a\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603178    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-home\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-home\") pod \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603212    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-dshm\") pod \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603259    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dddcb392-8e28-4cce-984c-1f06108f927a-tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb392-8e28-4cce-984c-1f06108f927a-tls-certs\") pod \"dddcb392-8e28-4cce-984c-1f06108f927a\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603322    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-home\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-home\") pod \"278c89c3-74ca-422d-b7c0-e13880664f59\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603376    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-dshm\") pod \"dddcb392-8e28-4cce-984c-1f06108f927a\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603380    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tmp-dir" pod "1bf42312-d61e-4b09-8a1b-37acd752ce5c" (UID: "1bf42312-d61e-4b09-8a1b-37acd752ce5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603429    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278c89c3-74ca-422d-b7c0-e13880664f59-tls-certs\" (UniqueName: \"kubernetes.io/secret/278c89c3-74ca-422d-b7c0-e13880664f59-tls-certs\") pod \"278c89c3-74ca-422d-b7c0-e13880664f59\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603457    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-tmp-dir\") pod \"dddcb392-8e28-4cce-984c-1f06108f927a\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603497    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278c89c3-74ca-422d-b7c0-e13880664f59-kube-api-access-tjc7q\" (UniqueName: \"kubernetes.io/projected/278c89c3-74ca-422d-b7c0-e13880664f59-kube-api-access-tjc7q\") pod \"278c89c3-74ca-422d-b7c0-e13880664f59\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603545    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-home\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-home\") pod \"dddcb392-8e28-4cce-984c-1f06108f927a\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603604    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-dshm\") pod \"278c89c3-74ca-422d-b7c0-e13880664f59\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603633    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-model-cache\") pod \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603666    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddcb392-8e28-4cce-984c-1f06108f927a-kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/dddcb392-8e28-4cce-984c-1f06108f927a-kube-api-access-gb722\") pod \"dddcb392-8e28-4cce-984c-1f06108f927a\" (UID: \"dddcb392-8e28-4cce-984c-1f06108f927a\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603720    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tls-certs\") pod \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\" (UID: \"1bf42312-d61e-4b09-8a1b-37acd752ce5c\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603801    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-tmp-dir\") pod \"278c89c3-74ca-422d-b7c0-e13880664f59\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.603843    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-model-cache\") pod \"278c89c3-74ca-422d-b7c0-e13880664f59\" (UID: \"278c89c3-74ca-422d-b7c0-e13880664f59\") "
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.604422    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-model-cache" pod "dddcb392-8e28-4cce-984c-1f06108f927a" (UID: "dddcb392-8e28-4cce-984c-1f06108f927a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.604497    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b61d8c13-0e9f-4f30-be94-cddeb5abb510-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.604518    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.604534    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6chm\" (UniqueName: \"kubernetes.io/projected/b61d8c13-0e9f-4f30-be94-cddeb5abb510-kube-api-access-x6chm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.604619    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-home" pod "1bf42312-d61e-4b09-8a1b-37acd752ce5c" (UID: "1bf42312-d61e-4b09-8a1b-37acd752ce5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.604745    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-tmp-dir" pod "dddcb392-8e28-4cce-984c-1f06108f927a" (UID: "dddcb392-8e28-4cce-984c-1f06108f927a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.604810    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-home" pod "278c89c3-74ca-422d-b7c0-e13880664f59" (UID: "278c89c3-74ca-422d-b7c0-e13880664f59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.607905    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-tmp-dir" pod "278c89c3-74ca-422d-b7c0-e13880664f59" (UID: "278c89c3-74ca-422d-b7c0-e13880664f59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.608442    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-home" pod "dddcb392-8e28-4cce-984c-1f06108f927a" (UID: "dddcb392-8e28-4cce-984c-1f06108f927a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.608648    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-model-cache" pod "1bf42312-d61e-4b09-8a1b-37acd752ce5c" (UID: "1bf42312-d61e-4b09-8a1b-37acd752ce5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.608876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.608830    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-model-cache" pod "278c89c3-74ca-422d-b7c0-e13880664f59" (UID: "278c89c3-74ca-422d-b7c0-e13880664f59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.616622 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.616572    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-dshm" pod "1bf42312-d61e-4b09-8a1b-37acd752ce5c" (UID: "1bf42312-d61e-4b09-8a1b-37acd752ce5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.617153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.617122    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278c89c3-74ca-422d-b7c0-e13880664f59-tls-certs" pod "278c89c3-74ca-422d-b7c0-e13880664f59" (UID: "278c89c3-74ca-422d-b7c0-e13880664f59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:02.617153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.617123    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddcb392-8e28-4cce-984c-1f06108f927a-kube-api-access-gb722" pod "dddcb392-8e28-4cce-984c-1f06108f927a" (UID: "dddcb392-8e28-4cce-984c-1f06108f927a"). InnerVolumeSpecName "kube-api-access-gb722". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:02.617308 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.617186    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dddcb392-8e28-4cce-984c-1f06108f927a-tls-certs" pod "dddcb392-8e28-4cce-984c-1f06108f927a" (UID: "dddcb392-8e28-4cce-984c-1f06108f927a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:02.617308 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.617246    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf42312-d61e-4b09-8a1b-37acd752ce5c-kube-api-access-rhhzg" pod "1bf42312-d61e-4b09-8a1b-37acd752ce5c" (UID: "1bf42312-d61e-4b09-8a1b-37acd752ce5c"). InnerVolumeSpecName "kube-api-access-rhhzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:02.617499 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.617470    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-dshm" pod "278c89c3-74ca-422d-b7c0-e13880664f59" (UID: "278c89c3-74ca-422d-b7c0-e13880664f59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.617642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.617624    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tls-certs" pod "1bf42312-d61e-4b09-8a1b-37acd752ce5c" (UID: "1bf42312-d61e-4b09-8a1b-37acd752ce5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:02.617872 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.617848    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-dshm" pod "dddcb392-8e28-4cce-984c-1f06108f927a" (UID: "dddcb392-8e28-4cce-984c-1f06108f927a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.618161 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.618139    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278c89c3-74ca-422d-b7c0-e13880664f59-kube-api-access-tjc7q" pod "278c89c3-74ca-422d-b7c0-e13880664f59" (UID: "278c89c3-74ca-422d-b7c0-e13880664f59"). InnerVolumeSpecName "kube-api-access-tjc7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:02.706483 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706380    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706483 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706429    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhzg\" (UniqueName: \"kubernetes.io/projected/1bf42312-d61e-4b09-8a1b-37acd752ce5c-kube-api-access-rhhzg\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706483 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706448    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706483 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706462    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706483 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706474    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706483 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706487    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb392-8e28-4cce-984c-1f06108f927a-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706499    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706510    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706521    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278c89c3-74ca-422d-b7c0-e13880664f59-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706533    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706546    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjc7q\" (UniqueName: \"kubernetes.io/projected/278c89c3-74ca-422d-b7c0-e13880664f59-kube-api-access-tjc7q\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706559    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dddcb392-8e28-4cce-984c-1f06108f927a-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706572    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706585    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf42312-d61e-4b09-8a1b-37acd752ce5c-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706598    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb722\" (UniqueName: \"kubernetes.io/projected/dddcb392-8e28-4cce-984c-1f06108f927a-kube-api-access-gb722\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706611    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf42312-d61e-4b09-8a1b-37acd752ce5c-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.706806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:02.706624    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278c89c3-74ca-422d-b7c0-e13880664f59-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.122473 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.122441    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 23:01:03.173765 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.173713    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" podUID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.37:50052\" within 1s: context deadline exceeded"
Oct 29 23:01:03.173965 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.173815    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 23:01:03.204355 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.204318    4742 generic.go:356] "Generic (PLEG): container finished" podID="278c89c3-74ca-422d-b7c0-e13880664f59" containerID="977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9" exitCode=0
Oct 29 23:01:03.204848 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.204386    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"
Oct 29 23:01:03.204848 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.204402    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" event={"ID":"278c89c3-74ca-422d-b7c0-e13880664f59","Type":"ContainerDied","Data":"977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9"}
Oct 29 23:01:03.204848 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.204452    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn" event={"ID":"278c89c3-74ca-422d-b7c0-e13880664f59","Type":"ContainerDied","Data":"330ad115bb77d1054083dc25b935fab343c51bc522435f76ffa3750e8ca96c7a"}
Oct 29 23:01:03.204848 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.204477    4742 scope.go:122] "RemoveContainer" containerID="977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9"
Oct 29 23:01:03.206405 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.206375    4742 generic.go:356] "Generic (PLEG): container finished" podID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerID="24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed" exitCode=0
Oct 29 23:01:03.206538 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.206482    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" event={"ID":"1bf42312-d61e-4b09-8a1b-37acd752ce5c","Type":"ContainerDied","Data":"24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed"}
Oct 29 23:01:03.206538 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.206514    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp" event={"ID":"1bf42312-d61e-4b09-8a1b-37acd752ce5c","Type":"ContainerDied","Data":"480645aeacbea5987a9e13d6456e9f326431be39b88d75f26cdd02861478cf66"}
Oct 29 23:01:03.206655 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.206609    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"
Oct 29 23:01:03.208920 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.208887    4742 generic.go:356] "Generic (PLEG): container finished" podID="dddcb392-8e28-4cce-984c-1f06108f927a" containerID="7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6" exitCode=0
Oct 29 23:01:03.209057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.208955    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"
Oct 29 23:01:03.209057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.208968    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" event={"ID":"dddcb392-8e28-4cce-984c-1f06108f927a","Type":"ContainerDied","Data":"7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6"}
Oct 29 23:01:03.209057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.209008    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp" event={"ID":"dddcb392-8e28-4cce-984c-1f06108f927a","Type":"ContainerDied","Data":"1fc4935eaa0d2e8098414d4bcc8ed7a4ab4a0e4b1137c0e496cfc75b71f97370"}
Oct 29 23:01:03.213425 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.213391    4742 generic.go:356] "Generic (PLEG): container finished" podID="2e961f1d-5b31-4639-98a8-018c1060b806" containerID="bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb" exitCode=0
Oct 29 23:01:03.213575 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.213430    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8" event={"ID":"2e961f1d-5b31-4639-98a8-018c1060b806","Type":"ContainerDied","Data":"bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb"}
Oct 29 23:01:03.213575 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.213494    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8" event={"ID":"2e961f1d-5b31-4639-98a8-018c1060b806","Type":"ContainerDied","Data":"24c787757d65b51776188b8b85f6edefbc514cb157649143618dfb122f2ebf35"}
Oct 29 23:01:03.213575 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.213537    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e961f1d-5b31-4639-98a8-018c1060b806-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e961f1d-5b31-4639-98a8-018c1060b806-plugins-config-volume\") pod \"2e961f1d-5b31-4639-98a8-018c1060b806\" (UID: \"2e961f1d-5b31-4639-98a8-018c1060b806\") "
Oct 29 23:01:03.213575 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.213551    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"
Oct 29 23:01:03.213793 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.213732    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e961f1d-5b31-4639-98a8-018c1060b806-kube-api-access-2pv9v\" (UniqueName: \"kubernetes.io/projected/2e961f1d-5b31-4639-98a8-018c1060b806-kube-api-access-2pv9v\") pod \"2e961f1d-5b31-4639-98a8-018c1060b806\" (UID: \"2e961f1d-5b31-4639-98a8-018c1060b806\") "
Oct 29 23:01:03.213921 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.213897    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e961f1d-5b31-4639-98a8-018c1060b806-plugins-config-volume" pod "2e961f1d-5b31-4639-98a8-018c1060b806" (UID: "2e961f1d-5b31-4639-98a8-018c1060b806"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:03.214528 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.214114    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e961f1d-5b31-4639-98a8-018c1060b806-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.215387 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.215364    4742 generic.go:356] "Generic (PLEG): container finished" podID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" containerID="b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8" exitCode=0
Oct 29 23:01:03.215491 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.215434    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"
Oct 29 23:01:03.215491 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.215448    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" event={"ID":"b61d8c13-0e9f-4f30-be94-cddeb5abb510","Type":"ContainerDied","Data":"b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8"}
Oct 29 23:01:03.215579 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.215502    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5" event={"ID":"b61d8c13-0e9f-4f30-be94-cddeb5abb510","Type":"ContainerDied","Data":"26c8920379efcd29ebc1c5e6de93d1055497b44e3627f97d86cc4bdbd2148def"}
Oct 29 23:01:03.216196 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.216172    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e961f1d-5b31-4639-98a8-018c1060b806-kube-api-access-2pv9v" pod "2e961f1d-5b31-4639-98a8-018c1060b806" (UID: "2e961f1d-5b31-4639-98a8-018c1060b806"). InnerVolumeSpecName "kube-api-access-2pv9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.217427 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.217413    4742 scope.go:122] "RemoveContainer" containerID="977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9"
Oct 29 23:01:03.217743 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:03.217719    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9\": container with ID starting with 977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9 not found: ID does not exist" containerID="977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9"
Oct 29 23:01:03.217819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.217753    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9"} err="failed to get container status \"977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9\": rpc error: code = NotFound desc = could not find container \"977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9\": container with ID starting with 977f15c9d92d96cd7fc3a7f049cc39732018a0cb3009bb54314988b6a78a56b9 not found: ID does not exist"
Oct 29 23:01:03.217819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.217783    4742 scope.go:122] "RemoveContainer" containerID="24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed"
Oct 29 23:01:03.233412 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.233252    4742 scope.go:122] "RemoveContainer" containerID="24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed"
Oct 29 23:01:03.233635 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:03.233608    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed\": container with ID starting with 24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed not found: ID does not exist" containerID="24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed"
Oct 29 23:01:03.233682 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.233652    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed"} err="failed to get container status \"24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed\": rpc error: code = NotFound desc = could not find container \"24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed\": container with ID starting with 24b2ce949373c4d6e1f460a5df1c4a6032f29a0bc41aa4100b2fa1be953dd3ed not found: ID does not exist"
Oct 29 23:01:03.233741 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.233681    4742 scope.go:122] "RemoveContainer" containerID="7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6"
Oct 29 23:01:03.244147 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.244124    4742 scope.go:122] "RemoveContainer" containerID="7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6"
Oct 29 23:01:03.244515 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:03.244492    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6\": container with ID starting with 7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6 not found: ID does not exist" containerID="7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6"
Oct 29 23:01:03.244675 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.244523    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6"} err="failed to get container status \"7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6\": rpc error: code = NotFound desc = could not find container \"7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6\": container with ID starting with 7ba679f86444e9a98151312bfdba9f0ebef46fb34db4e63b9a0dada4537b89b6 not found: ID does not exist"
Oct 29 23:01:03.244675 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.244542    4742 scope.go:122] "RemoveContainer" containerID="bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb"
Oct 29 23:01:03.254249 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.254223    4742 scope.go:122] "RemoveContainer" containerID="bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb"
Oct 29 23:01:03.254570 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:03.254550    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb\": container with ID starting with bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb not found: ID does not exist" containerID="bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb"
Oct 29 23:01:03.254639 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.254578    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb"} err="failed to get container status \"bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb\": rpc error: code = NotFound desc = could not find container \"bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb\": container with ID starting with bb04b9a5bc76cf36cee5f746fd2832d5723ef9605ff67699a24ab3be4edb3fbb not found: ID does not exist"
Oct 29 23:01:03.254639 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.254596    4742 scope.go:122] "RemoveContainer" containerID="b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8"
Oct 29 23:01:03.264576 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.264524    4742 scope.go:122] "RemoveContainer" containerID="b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8"
Oct 29 23:01:03.264924 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:03.264898    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8\": container with ID starting with b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8 not found: ID does not exist" containerID="b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8"
Oct 29 23:01:03.265024 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.264932    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8"} err="failed to get container status \"b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8\": rpc error: code = NotFound desc = could not find container \"b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8\": container with ID starting with b1ae7644e7597a6b25b5c611dcb5ecca14211874dce79e4feeedf2cf917bcfb8 not found: ID does not exist"
Oct 29 23:01:03.290192 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.290155    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"]
Oct 29 23:01:03.302648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.302613    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/simulator-w3-07eef2-kserve-65576c47f-2gcpn"]
Oct 29 23:01:03.315318 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.315280    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pv9v\" (UniqueName: \"kubernetes.io/projected/2e961f1d-5b31-4639-98a8-018c1060b806-kube-api-access-2pv9v\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.343525 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.343479    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"]
Oct 29 23:01:03.354776 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.354740    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-w3-07eef2-kserve-d6b6d96d5-hdtqp"]
Oct 29 23:01:03.400156 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.400117    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"]
Oct 29 23:01:03.409183 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.409045    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/distinct-2-w3-07eef2-kserve-7d46c6fddb-bprkp"]
Oct 29 23:01:03.475245 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.475210    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"]
Oct 29 23:01:03.485777 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.485740    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-20f88f-679fdf9bdc-x57m5"]
Oct 29 23:01:03.563686 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.563657    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"]
Oct 29 23:01:03.570727 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.570689    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-81f92d-65c9c79c65-7t4f8"]
Oct 29 23:01:03.816909 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.816882    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 23:01:03.924472 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924434    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-kube-api-access-rm9dl\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-kube-api-access-rm9dl\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.924653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924513    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-certs\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.924653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924577    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-socket\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.924744 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924662    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-credential-socket\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.924744 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924688    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-envoy\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.924744 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924730    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-podinfo\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.924907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924787    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57f7435-1e6b-4e25-af1a-c399471ca25d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f7435-1e6b-4e25-af1a-c399471ca25d-istiod-ca-cert\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.924907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924838    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-token\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-token\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.924907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924858    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-certs" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.924907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924873    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-socket" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.924907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.924869    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-data\") pod \"e57f7435-1e6b-4e25-af1a-c399471ca25d\" (UID: \"e57f7435-1e6b-4e25-af1a-c399471ca25d\") "
Oct 29 23:01:03.925216 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.925192    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-data" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.925272 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.925216    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-credential-socket" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.925332 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.925299    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57f7435-1e6b-4e25-af1a-c399471ca25d-istiod-ca-cert" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:03.925577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.925558    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.925648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.925582    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.925648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.925596    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.925648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.925608    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.925648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.925623    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f7435-1e6b-4e25-af1a-c399471ca25d-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.926909 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.926884    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-kube-api-access-rm9dl" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "kube-api-access-rm9dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.927015 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.926903    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-token" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.927015 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.926929    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-envoy" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.927155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:03.927137    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-podinfo" pod "e57f7435-1e6b-4e25-af1a-c399471ca25d" (UID: "e57f7435-1e6b-4e25-af1a-c399471ca25d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:04.027325 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.027291    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:04.027325 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.027322    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9dl\" (UniqueName: \"kubernetes.io/projected/e57f7435-1e6b-4e25-af1a-c399471ca25d-kube-api-access-rm9dl\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:04.027325 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.027332    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:04.027583 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.027341    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f7435-1e6b-4e25-af1a-c399471ca25d-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:04.193021 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.192980    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4" podUID="f952eacf-1713-4dba-890b-ac9838b1636b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:04.193224 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.193045    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr" podUID="33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:04.226325 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.226286    4742 generic.go:356] "Generic (PLEG): container finished" podID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerID="610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5" exitCode=0
Oct 29 23:01:04.226915 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.226364    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"
Oct 29 23:01:04.226915 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.226374    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" event={"ID":"e57f7435-1e6b-4e25-af1a-c399471ca25d","Type":"ContainerDied","Data":"610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5"}
Oct 29 23:01:04.226915 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.226473    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b" event={"ID":"e57f7435-1e6b-4e25-af1a-c399471ca25d","Type":"ContainerDied","Data":"fc1b3585fa6c692af22b0dbe4d7f02d0c48f17803d0575a860acb67ba5c620e8"}
Oct 29 23:01:04.226915 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.226538    4742 scope.go:122] "RemoveContainer" containerID="610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5"
Oct 29 23:01:04.241759 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.241732    4742 scope.go:122] "RemoveContainer" containerID="db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82"
Oct 29 23:01:04.252116 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.252095    4742 scope.go:122] "RemoveContainer" containerID="610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5"
Oct 29 23:01:04.252501 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:04.252479    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5\": container with ID starting with 610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5 not found: ID does not exist" containerID="610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5"
Oct 29 23:01:04.252575 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.252511    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5"} err="failed to get container status \"610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5\": rpc error: code = NotFound desc = could not find container \"610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5\": container with ID starting with 610e7041eb77946dc6d77ee9503cb964815e75ef5243c8c937bd3e21eddc64b5 not found: ID does not exist"
Oct 29 23:01:04.252575 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.252530    4742 scope.go:122] "RemoveContainer" containerID="db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82"
Oct 29 23:01:04.252859 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:04.252840    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82\": container with ID starting with db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82 not found: ID does not exist" containerID="db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82"
Oct 29 23:01:04.252914 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.252865    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82"} err="failed to get container status \"db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82\": rpc error: code = NotFound desc = could not find container \"db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82\": container with ID starting with db623a50fb12d3c84df5b97d01e6d2b4a7842a1c6ed80d076da4719003155a82 not found: ID does not exist"
Oct 29 23:01:04.294060 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.294025    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"]
Oct 29 23:01:04.299516 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.299486    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-xg97b"]
Oct 29 23:01:04.993071 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.993034    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" path="/var/lib/kubelet/pods/1bf42312-d61e-4b09-8a1b-37acd752ce5c/volumes"
Oct 29 23:01:04.993529 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.993514    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" path="/var/lib/kubelet/pods/278c89c3-74ca-422d-b7c0-e13880664f59/volumes"
Oct 29 23:01:04.993853 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.993841    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e961f1d-5b31-4639-98a8-018c1060b806" path="/var/lib/kubelet/pods/2e961f1d-5b31-4639-98a8-018c1060b806/volumes"
Oct 29 23:01:04.994481 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.994466    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" path="/var/lib/kubelet/pods/b61d8c13-0e9f-4f30-be94-cddeb5abb510/volumes"
Oct 29 23:01:04.994857 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.994845    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" path="/var/lib/kubelet/pods/dddcb392-8e28-4cce-984c-1f06108f927a/volumes"
Oct 29 23:01:04.995288 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:04.995275    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" path="/var/lib/kubelet/pods/e57f7435-1e6b-4e25-af1a-c399471ca25d/volumes"
Oct 29 23:01:05.757295 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:05.757215    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"]
Oct 29 23:01:05.757698 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:05.757427    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy" containerID="cri-o://946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561" gracePeriod=30
Oct 29 23:01:05.767478 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:05.767450    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"]
Oct 29 23:01:05.767665 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:05.767643    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy" containerID="cri-o://9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43" gracePeriod=30
Oct 29 23:01:05.830771 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:05.830738    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"]
Oct 29 23:01:05.830962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:05.830934    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" containerID="cri-o://f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45" gracePeriod=30
Oct 29 23:01:05.846776 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:05.846737    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"]
Oct 29 23:01:05.846960 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:05.846938    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy" containerID="cri-o://04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611" gracePeriod=30
Oct 29 23:01:09.230975 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.230936    4742 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:09.231364 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.231004    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:09.446117 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.446056    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"]
Oct 29 23:01:09.446306 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.446285    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" containerID="cri-o://77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4" gracePeriod=30
Oct 29 23:01:09.457209 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.457175    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"]
Oct 29 23:01:09.457420 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.457374    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy" containerID="cri-o://db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36" gracePeriod=30
Oct 29 23:01:09.541846 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.541756    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"]
Oct 29 23:01:09.542030 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.542007    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy" containerID="cri-o://613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349" gracePeriod=30
Oct 29 23:01:09.881587 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.881546    4742 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:09.881755 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:09.881616    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:11.078540 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.078465    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 23:01:11.148374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.148341    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 23:01:11.204055 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.204025    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 23:01:11.216221 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.216191    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 23:01:11.223329 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223303    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-credential-socket\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.223408 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223345    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-token\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.223408 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223373    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istiod-ca-cert\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.223552 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223460    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-token\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-token\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.223552 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223501    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-podinfo\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.223552 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223532    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-credential-socket" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.223703 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223581    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-data\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.223703 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223610    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-envoy\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.223703 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223637    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-podinfo\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.223703 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223671    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-data\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.223914 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223752    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-certs\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.223914 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223799    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-socket\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.223914 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223837    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istiod-ca-cert\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.223914 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223872    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-envoy\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.223914 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223871    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-data" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.223914 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223899    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istiod-ca-cert" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.224230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.223907    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-kube-api-access-gq6h4\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-kube-api-access-gq6h4\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.224230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224073    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-certs\") pod \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\" (UID: \"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2\") "
Oct 29 23:01:11.224230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224097    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-socket" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.224230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224169    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-kube-api-access-2cds7\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-kube-api-access-2cds7\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.224230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224214    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-credential-socket\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.224490 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224049    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-data" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.224490 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224255    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-socket\") pod \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\" (UID: \"bcc9dad8-a255-4d7a-b627-b4ac73b8b615\") "
Oct 29 23:01:11.224490 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224289    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-certs" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.224865 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224840    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istiod-ca-cert" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.224865 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224855    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.225007 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224877    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.225007 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224892    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.225007 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224906    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.225007 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224921    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.225007 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.224934    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.225280 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.225097    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-credential-socket" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.225280 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.225203    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-socket" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.225280 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.225235    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-certs" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.226254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.226226    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-podinfo" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.226397 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.226275    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-token" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.226789 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.226762    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-kube-api-access-2cds7" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "kube-api-access-2cds7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.226789 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.226777    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-envoy" pod "bcc9dad8-a255-4d7a-b627-b4ac73b8b615" (UID: "bcc9dad8-a255-4d7a-b627-b4ac73b8b615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.226951 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.226885    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-token" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.226951 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.226905    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-kube-api-access-gq6h4" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "kube-api-access-gq6h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.227232 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.227216    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-podinfo" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.227338 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.227321    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-envoy" pod "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" (UID: "28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.292790 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.292697    4742 generic.go:356] "Generic (PLEG): container finished" podID="692fbafb-57c8-4794-a6c4-c159075ae618" containerID="04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611" exitCode=0
Oct 29 23:01:11.292994 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.292756    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" event={"ID":"692fbafb-57c8-4794-a6c4-c159075ae618","Type":"ContainerDied","Data":"04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611"}
Oct 29 23:01:11.292994 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.292872    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"
Oct 29 23:01:11.292994 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.292941    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt" event={"ID":"692fbafb-57c8-4794-a6c4-c159075ae618","Type":"ContainerDied","Data":"1ec12a69c81a587ad6080ff6b11cd629eaa4cd3b91ddffe12b99d2c8cf6105d5"}
Oct 29 23:01:11.292994 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.292976    4742 scope.go:122] "RemoveContainer" containerID="04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611"
Oct 29 23:01:11.296078 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.296044    4742 generic.go:356] "Generic (PLEG): container finished" podID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerID="9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43" exitCode=0
Oct 29 23:01:11.296325 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.296279    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" event={"ID":"bcc9dad8-a255-4d7a-b627-b4ac73b8b615","Type":"ContainerDied","Data":"9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43"}
Oct 29 23:01:11.296389 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.296337    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l" event={"ID":"bcc9dad8-a255-4d7a-b627-b4ac73b8b615","Type":"ContainerDied","Data":"931294f10520f6adfda3beceea02f95876d6c4f8091c5450f03c9c6c699e361f"}
Oct 29 23:01:11.296389 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.296384    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"
Oct 29 23:01:11.307232 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.306777    4742 generic.go:356] "Generic (PLEG): container finished" podID="060de561-99f2-4378-b674-47d1e223a1c8" containerID="f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45" exitCode=0
Oct 29 23:01:11.307232 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.306844    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"
Oct 29 23:01:11.307232 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.306883    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" event={"ID":"060de561-99f2-4378-b674-47d1e223a1c8","Type":"ContainerDied","Data":"f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45"}
Oct 29 23:01:11.307232 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.306930    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z" event={"ID":"060de561-99f2-4378-b674-47d1e223a1c8","Type":"ContainerDied","Data":"1b17ac7ca6cd805154414d9ab87c69309f739040d60ad86c5c27d0ef74d1e354"}
Oct 29 23:01:11.320133 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.316278    4742 scope.go:122] "RemoveContainer" containerID="04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611"
Oct 29 23:01:11.320580 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:11.320533    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611\": container with ID starting with 04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611 not found: ID does not exist" containerID="04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611"
Oct 29 23:01:11.320759 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.320581    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611"} err="failed to get container status \"04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611\": rpc error: code = NotFound desc = could not find container \"04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611\": container with ID starting with 04dcbd97c19dd34f0de52c7055b341b882f390aaa61891d8f34026f5dad5d611 not found: ID does not exist"
Oct 29 23:01:11.320759 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.320602    4742 scope.go:122] "RemoveContainer" containerID="9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43"
Oct 29 23:01:11.320759 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.320695    4742 generic.go:356] "Generic (PLEG): container finished" podID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerID="946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561" exitCode=0
Oct 29 23:01:11.320759 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.320744    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" event={"ID":"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2","Type":"ContainerDied","Data":"946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561"}
Oct 29 23:01:11.320759 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.320759    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"
Oct 29 23:01:11.321286 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.320798    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs" event={"ID":"28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2","Type":"ContainerDied","Data":"32d830dfce82f2337cbdeeab97b0db9d21bf940238e94da695663d0e94e00b91"}
Oct 29 23:01:11.325437 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325406    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-envoy\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.325564 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325485    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-istio-token\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-istio-token\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.325564 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325517    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-credential-socket\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.325564 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325551    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060de561-99f2-4378-b674-47d1e223a1c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060de561-99f2-4378-b674-47d1e223a1c8-istiod-ca-cert\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.325757 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325593    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/060de561-99f2-4378-b674-47d1e223a1c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060de561-99f2-4378-b674-47d1e223a1c8-istio-podinfo\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.325757 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325664    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-data\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.325757 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325699    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-socket\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325773    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692fbafb-57c8-4794-a6c4-c159075ae618-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692fbafb-57c8-4794-a6c4-c159075ae618-istiod-ca-cert\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325859    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-envoy\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325933    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-kube-api-access-vbdkq\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-kube-api-access-vbdkq\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.325996    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-istio-token\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-istio-token\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326046    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-certs\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326076    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-credential-socket\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326137    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-kube-api-access-cwfwn\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-kube-api-access-cwfwn\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326150    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060de561-99f2-4378-b674-47d1e223a1c8-istiod-ca-cert" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326220    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-certs\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326257    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-socket\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.326317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326288    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-data\") pod \"060de561-99f2-4378-b674-47d1e223a1c8\" (UID: \"060de561-99f2-4378-b674-47d1e223a1c8\") "
Oct 29 23:01:11.326867 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326342    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/692fbafb-57c8-4794-a6c4-c159075ae618-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692fbafb-57c8-4794-a6c4-c159075ae618-istio-podinfo\") pod \"692fbafb-57c8-4794-a6c4-c159075ae618\" (UID: \"692fbafb-57c8-4794-a6c4-c159075ae618\") "
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326920    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326948    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326962    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326978    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.326991    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.327007    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.327020    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/060de561-99f2-4378-b674-47d1e223a1c8-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.327035    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq6h4\" (UniqueName: \"kubernetes.io/projected/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-kube-api-access-gq6h4\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.327049    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.327064    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cds7\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-kube-api-access-2cds7\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.327079    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.327112    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.327404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.327126    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9dad8-a255-4d7a-b627-b4ac73b8b615-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.328840 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.328422    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-data" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.329158 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.329130    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-certs" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.329390 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.329329    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-credential-socket" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.329475 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.329418    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-credential-socket" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.329754 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.329731    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692fbafb-57c8-4794-a6c4-c159075ae618-istiod-ca-cert" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.329835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.329741    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-socket" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.330013 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.329985    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-data" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.331475 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.331451    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-certs" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.331571 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.331550    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-socket" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.334336 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.334295    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-kube-api-access-cwfwn" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "kube-api-access-cwfwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.334676 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.334656    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-kube-api-access-vbdkq" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "kube-api-access-vbdkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.334765 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.334732    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-istio-token" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.334807 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.334764    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/060de561-99f2-4378-b674-47d1e223a1c8-istio-podinfo" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.334895 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.334879    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-envoy" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.335155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.335138    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-envoy" pod "060de561-99f2-4378-b674-47d1e223a1c8" (UID: "060de561-99f2-4378-b674-47d1e223a1c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.335391 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.335375    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-istio-token" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.336472 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.336439    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/692fbafb-57c8-4794-a6c4-c159075ae618-istio-podinfo" pod "692fbafb-57c8-4794-a6c4-c159075ae618" (UID: "692fbafb-57c8-4794-a6c4-c159075ae618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.338189 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.338074    4742 scope.go:122] "RemoveContainer" containerID="9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43"
Oct 29 23:01:11.338584 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:11.338564    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43\": container with ID starting with 9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43 not found: ID does not exist" containerID="9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43"
Oct 29 23:01:11.338678 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.338599    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43"} err="failed to get container status \"9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43\": rpc error: code = NotFound desc = could not find container \"9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43\": container with ID starting with 9ef97d18f050ad509a9963bb754ca97e4108472e7d40e3f180a972e67320dd43 not found: ID does not exist"
Oct 29 23:01:11.338678 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.338629    4742 scope.go:122] "RemoveContainer" containerID="f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45"
Oct 29 23:01:11.352648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.352531    4742 scope.go:122] "RemoveContainer" containerID="f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45"
Oct 29 23:01:11.352993 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:11.352967    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45\": container with ID starting with f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45 not found: ID does not exist" containerID="f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45"
Oct 29 23:01:11.353133 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.353001    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45"} err="failed to get container status \"f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45\": rpc error: code = NotFound desc = could not find container \"f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45\": container with ID starting with f79bf0485866382babaeaf6b267a68f161913bf43ad9464a82cde6330183cd45 not found: ID does not exist"
Oct 29 23:01:11.353133 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.353022    4742 scope.go:122] "RemoveContainer" containerID="946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561"
Oct 29 23:01:11.366078 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.366053    4742 scope.go:122] "RemoveContainer" containerID="946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561"
Oct 29 23:01:11.366491 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:11.366465    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561\": container with ID starting with 946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561 not found: ID does not exist" containerID="946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561"
Oct 29 23:01:11.366601 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.366498    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561"} err="failed to get container status \"946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561\": rpc error: code = NotFound desc = could not find container \"946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561\": container with ID starting with 946baa73903adc5282a2de16e0e39f04b2cb1a59d2c3c83f791dd29fb38de561 not found: ID does not exist"
Oct 29 23:01:11.428729 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428698    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428729 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428726    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbdkq\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-kube-api-access-vbdkq\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428729 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428736    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428748    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428757    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428767    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwfwn\" (UniqueName: \"kubernetes.io/projected/692fbafb-57c8-4794-a6c4-c159075ae618-kube-api-access-cwfwn\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428776    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428784    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428792    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428799    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692fbafb-57c8-4794-a6c4-c159075ae618-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428816    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428825    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060de561-99f2-4378-b674-47d1e223a1c8-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428834    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428842    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/060de561-99f2-4378-b674-47d1e223a1c8-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428850    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692fbafb-57c8-4794-a6c4-c159075ae618-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428858    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/060de561-99f2-4378-b674-47d1e223a1c8-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.428982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.428866    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692fbafb-57c8-4794-a6c4-c159075ae618-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.438724 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.438690    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"]
Oct 29 23:01:11.455952 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.455917    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-ls59l"]
Oct 29 23:01:11.578702 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.578666    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"]
Oct 29 23:01:11.585835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.585808    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-snjfs"]
Oct 29 23:01:11.676269 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.676229    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"]
Oct 29 23:01:11.684391 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.684355    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-9vzwt"]
Oct 29 23:01:11.746030 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.745995    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"]
Oct 29 23:01:11.752203 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:11.752171    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-4wp2z"]
Oct 29 23:01:12.990779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:12.990742    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060de561-99f2-4378-b674-47d1e223a1c8" path="/var/lib/kubelet/pods/060de561-99f2-4378-b674-47d1e223a1c8/volumes"
Oct 29 23:01:12.991439 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:12.991404    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" path="/var/lib/kubelet/pods/28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2/volumes"
Oct 29 23:01:12.991959 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:12.991943    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" path="/var/lib/kubelet/pods/692fbafb-57c8-4794-a6c4-c159075ae618/volumes"
Oct 29 23:01:12.992754 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:12.992737    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" path="/var/lib/kubelet/pods/bcc9dad8-a255-4d7a-b627-b4ac73b8b615/volumes"
Oct 29 23:01:14.192630 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.192591    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4" podUID="f952eacf-1713-4dba-890b-ac9838b1636b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:14.193059 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.192633    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr" podUID="33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:14.751133 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.751106    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 23:01:14.787806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.787761    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 23:01:14.836924 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.836896    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 23:01:14.869450 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869409    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26438b13-81c6-4c88-a88e-19bceef48774-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26438b13-81c6-4c88-a88e-19bceef48774-istiod-ca-cert\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.869602 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869466    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-envoy\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.869602 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869521    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-certs\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.869602 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869559    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4124490-af35-452c-8b90-2d878d194412-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4124490-af35-452c-8b90-2d878d194412-istiod-ca-cert\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.869602 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869591    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-credential-socket\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.869820 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869618    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-istio-token\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-istio-token\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.869820 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869649    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-kube-api-access-csrnp\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-kube-api-access-csrnp\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.869820 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869731    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-envoy\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.870012 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869834    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-certs" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.870012 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.869877    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-credential-socket\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.870171 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.870138    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-data\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.870253 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.870172    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4124490-af35-452c-8b90-2d878d194412-istiod-ca-cert" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:14.871078 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.870851    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-data" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.871078 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.871023    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-credential-socket" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.871313 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.871283    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26438b13-81c6-4c88-a88e-19bceef48774-istiod-ca-cert" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:14.872404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.872355    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-kube-api-access-csrnp" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "kube-api-access-csrnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.872528 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.870196    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26438b13-81c6-4c88-a88e-19bceef48774-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26438b13-81c6-4c88-a88e-19bceef48774-istio-podinfo\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.872580 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.872553    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-kube-api-access-xg5md\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-kube-api-access-xg5md\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.872699 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.872664    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4124490-af35-452c-8b90-2d878d194412-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4124490-af35-452c-8b90-2d878d194412-istio-podinfo\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.872785 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.872769    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-data\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.872878 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.872857    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-certs\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.872957 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.872938    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-istio-token\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-istio-token\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.873014 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.873005    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-socket\") pod \"b4124490-af35-452c-8b90-2d878d194412\" (UID: \"b4124490-af35-452c-8b90-2d878d194412\") "
Oct 29 23:01:14.873074 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.873061    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-socket\") pod \"26438b13-81c6-4c88-a88e-19bceef48774\" (UID: \"26438b13-81c6-4c88-a88e-19bceef48774\") "
Oct 29 23:01:14.874264 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.873738    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-data" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.875418 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.875393    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.875562 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.875424    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4124490-af35-452c-8b90-2d878d194412-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.875562 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.875441    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.875562 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.875466    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrnp\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-kube-api-access-csrnp\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.875562 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.875483    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.875562 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.875499    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.877127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.875672    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26438b13-81c6-4c88-a88e-19bceef48774-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.877127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.876009    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-socket" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.877127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.876433    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-socket" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.877127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.876768    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-certs" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.877127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.876892    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-credential-socket" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.886172 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.886128    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4124490-af35-452c-8b90-2d878d194412-istio-podinfo" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:14.886808 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.886766    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-kube-api-access-xg5md" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "kube-api-access-xg5md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.887071 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.887036    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-envoy" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.887208 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.887128    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-envoy" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.887983 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.887927    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-istio-token" pod "b4124490-af35-452c-8b90-2d878d194412" (UID: "b4124490-af35-452c-8b90-2d878d194412"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.889480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.889452    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-istio-token" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.889562 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.889539    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26438b13-81c6-4c88-a88e-19bceef48774-istio-podinfo" pod "26438b13-81c6-4c88-a88e-19bceef48774" (UID: "26438b13-81c6-4c88-a88e-19bceef48774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:14.976974 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.976868    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-token\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-token\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977172 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.976988    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-data\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977172 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977066    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istiod-ca-cert\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977172 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977144    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-credential-socket\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977355 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977177    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-kube-api-access-x7j6p\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-kube-api-access-x7j6p\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977355 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977212    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-envoy\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977355 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977250    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-certs\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977355 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977304    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-podinfo\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977355 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977329    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-data" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.977355 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977346    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-socket\") pod \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\" (UID: \"8b3eee93-3339-4aab-aa07-b7afcf5a56ca\") "
Oct 29 23:01:14.977720 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977530    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-credential-socket" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.977720 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977544    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-certs" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.977853 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977810    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-socket" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.977911 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977868    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4124490-af35-452c-8b90-2d878d194412-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.977911 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977890    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.977911 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977904    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977919    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977935    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977949    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26438b13-81c6-4c88-a88e-19bceef48774-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977964    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977978    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5md\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-kube-api-access-xg5md\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.977992    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4124490-af35-452c-8b90-2d878d194412-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.978006    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.978020    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26438b13-81c6-4c88-a88e-19bceef48774-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.978034    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.978048    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26438b13-81c6-4c88-a88e-19bceef48774-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978555 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.978063    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4124490-af35-452c-8b90-2d878d194412-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.978555 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.978274    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istiod-ca-cert" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:14.979384 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.979360    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-token" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.980005 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.979982    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-envoy" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.980109 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.980059    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-kube-api-access-x7j6p" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "kube-api-access-x7j6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.980109 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:14.980064    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-podinfo" pod "8b3eee93-3339-4aab-aa07-b7afcf5a56ca" (UID: "8b3eee93-3339-4aab-aa07-b7afcf5a56ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:15.079607 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.079572    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.079607 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.079603    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.079819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.079618    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.079819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.079631    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.079819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.079643    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7j6p\" (UniqueName: \"kubernetes.io/projected/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-kube-api-access-x7j6p\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.079819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.079655    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3eee93-3339-4aab-aa07-b7afcf5a56ca-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.356775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.356739    4742 generic.go:356] "Generic (PLEG): container finished" podID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerID="613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349" exitCode=0
Oct 29 23:01:15.357309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.356827    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"
Oct 29 23:01:15.357309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.356933    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" event={"ID":"8b3eee93-3339-4aab-aa07-b7afcf5a56ca","Type":"ContainerDied","Data":"613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349"}
Oct 29 23:01:15.357309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.356977    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt" event={"ID":"8b3eee93-3339-4aab-aa07-b7afcf5a56ca","Type":"ContainerDied","Data":"0d49cc8561a281a1f5ed2d2f3cfe7439b3afeeaaae70b38533ea4fbd7c773405"}
Oct 29 23:01:15.357309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.357002    4742 scope.go:122] "RemoveContainer" containerID="613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349"
Oct 29 23:01:15.361341 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.361307    4742 generic.go:356] "Generic (PLEG): container finished" podID="26438b13-81c6-4c88-a88e-19bceef48774" containerID="db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36" exitCode=0
Oct 29 23:01:15.361538 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.361388    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" event={"ID":"26438b13-81c6-4c88-a88e-19bceef48774","Type":"ContainerDied","Data":"db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36"}
Oct 29 23:01:15.361538 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.361472    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d" event={"ID":"26438b13-81c6-4c88-a88e-19bceef48774","Type":"ContainerDied","Data":"b854856c108525168be2eca99a0af6275a5404108d49e853c7cbec1dd42703bf"}
Oct 29 23:01:15.361538 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.361399    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"
Oct 29 23:01:15.363989 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.363962    4742 generic.go:356] "Generic (PLEG): container finished" podID="b4124490-af35-452c-8b90-2d878d194412" containerID="77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4" exitCode=0
Oct 29 23:01:15.364178 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.364158    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" event={"ID":"b4124490-af35-452c-8b90-2d878d194412","Type":"ContainerDied","Data":"77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4"}
Oct 29 23:01:15.364258 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.364219    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m" event={"ID":"b4124490-af35-452c-8b90-2d878d194412","Type":"ContainerDied","Data":"2776c9ae24a64294a140a9e41de265408a3fba62a75060db0f639dc00ded3588"}
Oct 29 23:01:15.364332 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.364318    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"
Oct 29 23:01:15.371834 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.371815    4742 scope.go:122] "RemoveContainer" containerID="613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349"
Oct 29 23:01:15.372290 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:15.372248    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349\": container with ID starting with 613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349 not found: ID does not exist" containerID="613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349"
Oct 29 23:01:15.372605 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.372293    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349"} err="failed to get container status \"613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349\": rpc error: code = NotFound desc = could not find container \"613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349\": container with ID starting with 613934d34acb172cc20d3d1c3d2a167d1219619b51ba562915fc639c36830349 not found: ID does not exist"
Oct 29 23:01:15.372605 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.372325    4742 scope.go:122] "RemoveContainer" containerID="db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36"
Oct 29 23:01:15.384749 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.384717    4742 scope.go:122] "RemoveContainer" containerID="db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36"
Oct 29 23:01:15.385296 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:15.385259    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36\": container with ID starting with db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36 not found: ID does not exist" containerID="db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36"
Oct 29 23:01:15.385568 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.385531    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36"} err="failed to get container status \"db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36\": rpc error: code = NotFound desc = could not find container \"db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36\": container with ID starting with db7f5d314f0a94afd8543bf5869cdae21c8304c3f36f10f264169ca9345bdf36 not found: ID does not exist"
Oct 29 23:01:15.385568 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.385569    4742 scope.go:122] "RemoveContainer" containerID="77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4"
Oct 29 23:01:15.401211 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.401190    4742 scope.go:122] "RemoveContainer" containerID="77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4"
Oct 29 23:01:15.401548 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:15.401527    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4\": container with ID starting with 77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4 not found: ID does not exist" containerID="77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4"
Oct 29 23:01:15.401615 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.401562    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4"} err="failed to get container status \"77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4\": rpc error: code = NotFound desc = could not find container \"77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4\": container with ID starting with 77b435d20c4ab475258121bc39fa3735a012b3881a2d3cf06b10e93a99041bd4 not found: ID does not exist"
Oct 29 23:01:15.445201 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.445154    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"]
Oct 29 23:01:15.511148 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.511115    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-k544d"]
Oct 29 23:01:15.563529 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.563494    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"]
Oct 29 23:01:15.579439 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.579405    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-4ms4m"]
Oct 29 23:01:15.660907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.660829    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"]
Oct 29 23:01:15.678118 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:15.678069    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-2swwt"]
Oct 29 23:01:16.132407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:16.132370    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"]
Oct 29 23:01:16.132622 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:16.132575    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" containerID="cri-o://290b21423a9e6505f52b16acb1e0bc1c49cdae1d84a720a6ae78956bb96ff1a4" gracePeriod=30
Oct 29 23:01:16.179015 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:16.178981    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"]
Oct 29 23:01:16.179238 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:16.179215    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" containerID="cri-o://224fae60d4c495b73f7101b3d7443fe92d31b7b37a21582e5c6c76ee88a06be8" gracePeriod=30
Oct 29 23:01:16.992254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:16.992219    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26438b13-81c6-4c88-a88e-19bceef48774" path="/var/lib/kubelet/pods/26438b13-81c6-4c88-a88e-19bceef48774/volumes"
Oct 29 23:01:16.992822 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:16.992796    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" path="/var/lib/kubelet/pods/8b3eee93-3339-4aab-aa07-b7afcf5a56ca/volumes"
Oct 29 23:01:16.993257 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:16.993241    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4124490-af35-452c-8b90-2d878d194412" path="/var/lib/kubelet/pods/b4124490-af35-452c-8b90-2d878d194412/volumes"
Oct 29 23:01:18.142984 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.142957    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 23:01:18.317783 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.317683    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f952eacf-1713-4dba-890b-ac9838b1636b-kube-api-access-6x4td\" (UniqueName: \"kubernetes.io/projected/f952eacf-1713-4dba-890b-ac9838b1636b-kube-api-access-6x4td\") pod \"f952eacf-1713-4dba-890b-ac9838b1636b\" (UID: \"f952eacf-1713-4dba-890b-ac9838b1636b\") "
Oct 29 23:01:18.317783 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.317733    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f952eacf-1713-4dba-890b-ac9838b1636b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952eacf-1713-4dba-890b-ac9838b1636b-plugins-config-volume\") pod \"f952eacf-1713-4dba-890b-ac9838b1636b\" (UID: \"f952eacf-1713-4dba-890b-ac9838b1636b\") "
Oct 29 23:01:18.318185 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.318158    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f952eacf-1713-4dba-890b-ac9838b1636b-plugins-config-volume" pod "f952eacf-1713-4dba-890b-ac9838b1636b" (UID: "f952eacf-1713-4dba-890b-ac9838b1636b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:18.319886 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.319864    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f952eacf-1713-4dba-890b-ac9838b1636b-kube-api-access-6x4td" pod "f952eacf-1713-4dba-890b-ac9838b1636b" (UID: "f952eacf-1713-4dba-890b-ac9838b1636b"). InnerVolumeSpecName "kube-api-access-6x4td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:18.400319 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.400282    4742 generic.go:356] "Generic (PLEG): container finished" podID="f952eacf-1713-4dba-890b-ac9838b1636b" containerID="7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24" exitCode=0
Oct 29 23:01:18.400471 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.400341    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"
Oct 29 23:01:18.400471 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.400373    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4" event={"ID":"f952eacf-1713-4dba-890b-ac9838b1636b","Type":"ContainerDied","Data":"7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24"}
Oct 29 23:01:18.400471 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.400468    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4" event={"ID":"f952eacf-1713-4dba-890b-ac9838b1636b","Type":"ContainerDied","Data":"e8f9177ac258f692a581f1d6422ac5e6be0a5f1d6fdd3b23aeb11a236fb3f45a"}
Oct 29 23:01:18.400593 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.400496    4742 scope.go:122] "RemoveContainer" containerID="7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24"
Oct 29 23:01:18.410464 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.410437    4742 scope.go:122] "RemoveContainer" containerID="7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24"
Oct 29 23:01:18.410788 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:18.410763    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24\": container with ID starting with 7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24 not found: ID does not exist" containerID="7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24"
Oct 29 23:01:18.410835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.410798    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24"} err="failed to get container status \"7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24\": rpc error: code = NotFound desc = could not find container \"7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24\": container with ID starting with 7c4c50c9a7b48ee30057bb60f6c4bda66c045e1a23b6a2f634605fcc7303ad24 not found: ID does not exist"
Oct 29 23:01:18.419752 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.419717    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x4td\" (UniqueName: \"kubernetes.io/projected/f952eacf-1713-4dba-890b-ac9838b1636b-kube-api-access-6x4td\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:18.419752 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.419754    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f952eacf-1713-4dba-890b-ac9838b1636b-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:18.476251 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.476216    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"]
Oct 29 23:01:18.487429 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.487398    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-07eef2-66b5879699-25ld4"]
Oct 29 23:01:18.493842 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.493810    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:18.493966 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.493877    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:18.541687 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.541646    4742 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:18.541889 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.541736    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:18.636147 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.636109    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 23:01:18.723831 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.723790    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-plugins-config-volume\") pod \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\" (UID: \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\") "
Oct 29 23:01:18.723981 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.723866    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-kube-api-access-nlltx\" (UniqueName: \"kubernetes.io/projected/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-kube-api-access-nlltx\") pod \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\" (UID: \"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23\") "
Oct 29 23:01:18.724200 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.724177    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-plugins-config-volume" pod "33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" (UID: "33bcb0b6-9f48-4a6e-bd40-111de6a3cf23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:18.724373 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.724348    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:18.725990 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.725965    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-kube-api-access-nlltx" pod "33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" (UID: "33bcb0b6-9f48-4a6e-bd40-111de6a3cf23"). InnerVolumeSpecName "kube-api-access-nlltx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:18.825864 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.825774    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlltx\" (UniqueName: \"kubernetes.io/projected/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23-kube-api-access-nlltx\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:18.989940 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:18.989909    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f952eacf-1713-4dba-890b-ac9838b1636b" path="/var/lib/kubelet/pods/f952eacf-1713-4dba-890b-ac9838b1636b/volumes"
Oct 29 23:01:19.408562 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.408527    4742 generic.go:356] "Generic (PLEG): container finished" podID="33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" containerID="a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa" exitCode=0
Oct 29 23:01:19.409029 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.408572    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"
Oct 29 23:01:19.409029 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.408608    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr" event={"ID":"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23","Type":"ContainerDied","Data":"a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa"}
Oct 29 23:01:19.409029 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.408663    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr" event={"ID":"33bcb0b6-9f48-4a6e-bd40-111de6a3cf23","Type":"ContainerDied","Data":"3a124bd43397f17dab22de4648d27af4613adacf36251ff9fe57aafba90ea6dc"}
Oct 29 23:01:19.409029 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.408694    4742 scope.go:122] "RemoveContainer" containerID="a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa"
Oct 29 23:01:19.418728 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.418703    4742 scope.go:122] "RemoveContainer" containerID="a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa"
Oct 29 23:01:19.419031 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:19.419012    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa\": container with ID starting with a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa not found: ID does not exist" containerID="a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa"
Oct 29 23:01:19.419167 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.419039    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa"} err="failed to get container status \"a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa\": rpc error: code = NotFound desc = could not find container \"a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa\": container with ID starting with a2129456d48d62eddb36db35ba4f3bfe3a7ee3f59711e3c66fcf1cad434d96aa not found: ID does not exist"
Oct 29 23:01:19.488965 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.488927    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"]
Oct 29 23:01:19.510761 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:19.510722    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-07eef2-5dc68d6bd9-8jlnr"]
Oct 29 23:01:20.991036 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:20.991004    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" path="/var/lib/kubelet/pods/33bcb0b6-9f48-4a6e-bd40-111de6a3cf23/volumes"
Oct 29 23:01:21.428984 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.428949    4742 generic.go:356] "Generic (PLEG): container finished" podID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerID="224fae60d4c495b73f7101b3d7443fe92d31b7b37a21582e5c6c76ee88a06be8" exitCode=0
Oct 29 23:01:21.429168 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.429041    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" event={"ID":"dc7fd15e-1f5c-407a-b86a-677851e1731b","Type":"ContainerDied","Data":"224fae60d4c495b73f7101b3d7443fe92d31b7b37a21582e5c6c76ee88a06be8"}
Oct 29 23:01:21.430879 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.430853    4742 generic.go:356] "Generic (PLEG): container finished" podID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerID="290b21423a9e6505f52b16acb1e0bc1c49cdae1d84a720a6ae78956bb96ff1a4" exitCode=0
Oct 29 23:01:21.431018 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.430891    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" event={"ID":"08f9d83d-a5a8-4739-9769-bdc9c275ab7a","Type":"ContainerDied","Data":"290b21423a9e6505f52b16acb1e0bc1c49cdae1d84a720a6ae78956bb96ff1a4"}
Oct 29 23:01:21.431018 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.430934    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n" event={"ID":"08f9d83d-a5a8-4739-9769-bdc9c275ab7a","Type":"ContainerDied","Data":"ecd14dabf7e940733cc0cd0a5111690ff9d53b7172c51a727ac8eff0407c8a6b"}
Oct 29 23:01:21.431018 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.430957    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecd14dabf7e940733cc0cd0a5111690ff9d53b7172c51a727ac8eff0407c8a6b"
Oct 29 23:01:21.435015 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.434992    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 23:01:21.465068 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465043    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-token\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-token\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465191 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465121    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-socket\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465267 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465206    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istiod-ca-cert\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465323 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465311    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-envoy\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465382 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465347    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-data\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465380    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-certs\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465418    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-kube-api-access-5lkkq\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-kube-api-access-5lkkq\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465550 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465474    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-socket" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.465550 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465501    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-podinfo\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465644 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465567    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-credential-socket\") pod \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\" (UID: \"08f9d83d-a5a8-4739-9769-bdc9c275ab7a\") "
Oct 29 23:01:21.465701 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465662    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istiod-ca-cert" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:21.465701 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465689    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-data" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.465958 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465912    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-certs" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.466050 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.465992    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.466050 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.466014    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.466050 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.466029    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.466259 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.466238    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-credential-socket" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.467472 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.467425    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-token" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:21.467819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.467798    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-envoy" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.467926 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.467868    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-podinfo" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:21.468168 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.468145    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-kube-api-access-5lkkq" pod "08f9d83d-a5a8-4739-9769-bdc9c275ab7a" (UID: "08f9d83d-a5a8-4739-9769-bdc9c275ab7a"). InnerVolumeSpecName "kube-api-access-5lkkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:21.486274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.486242    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 23:01:21.567025 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.566922    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-kube-api-access-l6gqf\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-kube-api-access-l6gqf\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567025 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.566980    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-certs\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567025 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567011    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-data\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567354 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567098    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-token\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-token\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567354 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567200    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7fd15e-1f5c-407a-b86a-677851e1731b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7fd15e-1f5c-407a-b86a-677851e1731b-istiod-ca-cert\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567354 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567227    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-socket\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567354 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567273    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-envoy\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567354 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567300    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-podinfo\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567354 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567346    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-credential-socket\") pod \"dc7fd15e-1f5c-407a-b86a-677851e1731b\" (UID: \"dc7fd15e-1f5c-407a-b86a-677851e1731b\") "
Oct 29 23:01:21.567650 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567445    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-certs" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.567750 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567715    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-socket" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.567750 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567735    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-credential-socket" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.567886 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567869    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.567949 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567893    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.567949 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567907    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.567949 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567922    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.567949 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567936    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lkkq\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-kube-api-access-5lkkq\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.568183 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567950    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.568183 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567964    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.568183 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567977    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.568183 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.567990    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f9d83d-a5a8-4739-9769-bdc9c275ab7a-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.568401 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.568378    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-data" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.568827 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.568795    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7fd15e-1f5c-407a-b86a-677851e1731b-istiod-ca-cert" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:21.570312 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.570288    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-token" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:21.570396 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.570322    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-kube-api-access-l6gqf" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "kube-api-access-l6gqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:21.570648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.570622    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-envoy" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.570689 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.570652    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-podinfo" pod "dc7fd15e-1f5c-407a-b86a-677851e1731b" (UID: "dc7fd15e-1f5c-407a-b86a-677851e1731b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:21.669409 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.669363    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7fd15e-1f5c-407a-b86a-677851e1731b-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.669409 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.669395    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.669409 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.669403    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.669409 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.669414    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6gqf\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-kube-api-access-l6gqf\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.669409 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.669422    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.669825 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:21.669430    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7fd15e-1f5c-407a-b86a-677851e1731b-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:22.439424 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.439349    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"
Oct 29 23:01:22.439424 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.439359    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p" event={"ID":"dc7fd15e-1f5c-407a-b86a-677851e1731b","Type":"ContainerDied","Data":"67436276cf670fdf8fa9efdf22835126b725b8b1221455e721bbfe679729a639"}
Oct 29 23:01:22.439896 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.439490    4742 scope.go:122] "RemoveContainer" containerID="224fae60d4c495b73f7101b3d7443fe92d31b7b37a21582e5c6c76ee88a06be8"
Oct 29 23:01:22.439896 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.439546    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"
Oct 29 23:01:22.521911 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.521877    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"]
Oct 29 23:01:22.549842 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.549802    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-fsg5p"]
Oct 29 23:01:22.597675 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.597640    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"]
Oct 29 23:01:22.612936 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.612903    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-5sd7n"]
Oct 29 23:01:22.992588 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.992559    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" path="/var/lib/kubelet/pods/08f9d83d-a5a8-4739-9769-bdc9c275ab7a/volumes"
Oct 29 23:01:22.993006 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:22.992992    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" path="/var/lib/kubelet/pods/dc7fd15e-1f5c-407a-b86a-677851e1731b/volumes"
Oct 29 23:01:23.123488 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:23.123453    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:01:23.306781 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:23.306679    4742 scope.go:122] "RemoveContainer" containerID="ad263e801eb5771d198a765dcc126724099b33474d65ba78017d0b30f4fc38a6"
Oct 29 23:01:23.316393 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:23.316371    4742 scope.go:122] "RemoveContainer" containerID="1ebb403fa4e892f801e01619b34be43f604604c4f4d533df4cb0b3757e7f563e"
Oct 29 23:01:23.325528 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:23.325503    4742 scope.go:122] "RemoveContainer" containerID="b67b21c45457484d74b7b5e602aa482995ddd181e888a3bb0e9f1f12151c5495"
Oct 29 23:01:23.335280 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:23.335258    4742 scope.go:122] "RemoveContainer" containerID="5433c076f8accb9ae0c7b1e70e20c4668a532e7e5b6926de9790af6eb1c8d880"
Oct 29 23:01:23.344284 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:23.344262    4742 scope.go:122] "RemoveContainer" containerID="0e136ea87b20413cffebdd99603a6cd30bafd7b13da0dd327aaab4203ee8ec11"
Oct 29 23:01:23.353593 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:23.353571    4742 scope.go:122] "RemoveContainer" containerID="e177045e26e70030e345e97dd884c8fb320646e2cb85675321c3886c3ada3611"
Oct 29 23:01:23.361988 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:23.361965    4742 scope.go:122] "RemoveContainer" containerID="feef846888f25ec00dc6ef585b79dc9ef1b9d682ae7d8c7c3a5a2ad59d555235"
Oct 29 23:01:24.897859 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:24.897827    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"]
Oct 29 23:01:24.898366 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:24.898062    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy" containerID="cri-o://794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330" gracePeriod=30
Oct 29 23:01:24.909327 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:24.909297    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"]
Oct 29 23:01:24.909550 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:24.909511    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy" containerID="cri-o://70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4" gracePeriod=30
Oct 29 23:01:24.957164 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:24.957125    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"]
Oct 29 23:01:24.957400 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:24.957375    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy" containerID="cri-o://41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f" gracePeriod=30
Oct 29 23:01:29.928511 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:29.928474    4742 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d97a9b0_9aed_43ea_90ed_d280767bc9cc.slice/crio-794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:01:30.225611 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.225579    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 23:01:30.246077 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.246048    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 23:01:30.262993 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.262963    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-kube-api-access-vmft4\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-kube-api-access-vmft4\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263158 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263014    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-istio-token\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-istio-token\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263158 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263061    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091431d4-cb1e-46ef-babb-e5f54e589476-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091431d4-cb1e-46ef-babb-e5f54e589476-istiod-ca-cert\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263158 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263151    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-certs\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263343 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263199    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-data\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263343 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263268    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-envoy\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263343 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263324    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-credential-socket\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263499 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263381    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/091431d4-cb1e-46ef-babb-e5f54e589476-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091431d4-cb1e-46ef-babb-e5f54e589476-istio-podinfo\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263499 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263427    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-socket\") pod \"091431d4-cb1e-46ef-babb-e5f54e589476\" (UID: \"091431d4-cb1e-46ef-babb-e5f54e589476\") "
Oct 29 23:01:30.263887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263617    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091431d4-cb1e-46ef-babb-e5f54e589476-istiod-ca-cert" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:30.263887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263741    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-certs" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.263887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263872    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-credential-socket" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.264110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263891    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-socket" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.264110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.263940    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-data" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.264110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.264016    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.264110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.264038    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.264110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.264054    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091431d4-cb1e-46ef-babb-e5f54e589476-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.264110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.264068    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.265683 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.265658    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-kube-api-access-vmft4" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "kube-api-access-vmft4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.265773 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.265702    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-istio-token" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.265954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.265939    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/091431d4-cb1e-46ef-babb-e5f54e589476-istio-podinfo" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:30.266131 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.266115    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-envoy" pod "091431d4-cb1e-46ef-babb-e5f54e589476" (UID: "091431d4-cb1e-46ef-babb-e5f54e589476"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.273493 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.273462    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 23:01:30.365429 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365390    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-data\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.365618 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365439    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-podinfo\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.365618 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365457    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-token\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-token\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.365618 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365483    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-socket\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.365618 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365528    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-certs\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.365618 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365572    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-credential-socket\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.365618 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365600    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-envoy\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365656    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-certs\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365715    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-data" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365728    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-token\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-token\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365816    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-data\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365827    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-socket" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365858    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56eace7-1ab2-4646-9298-9f652a46bfff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56eace7-1ab2-4646-9298-9f652a46bfff-istiod-ca-cert\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365902    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-socket\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365926    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-credential-socket" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.365956 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365944    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istiod-ca-cert\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.366463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.365988    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-envoy\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.366463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366029    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-certs" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.366463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366055    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-kube-api-access-z9bc7\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-kube-api-access-z9bc7\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.366463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366104    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-credential-socket\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.366463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366147    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-kube-api-access-h4rld\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-kube-api-access-h4rld\") pod \"c56eace7-1ab2-4646-9298-9f652a46bfff\" (UID: \"c56eace7-1ab2-4646-9298-9f652a46bfff\") "
Oct 29 23:01:30.366463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366195    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-podinfo\") pod \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\" (UID: \"3d97a9b0-9aed-43ea-90ed-d280767bc9cc\") "
Oct 29 23:01:30.366463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366332    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56eace7-1ab2-4646-9298-9f652a46bfff-istiod-ca-cert" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:30.366804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366472    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-data" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.366804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366179    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-certs" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.366804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366733    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091431d4-cb1e-46ef-babb-e5f54e589476-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.366804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366752    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.366804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366753    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-credential-socket" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.366804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366767    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56eace7-1ab2-4646-9298-9f652a46bfff-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.366804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366786    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-socket" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.366804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366798    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmft4\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-kube-api-access-vmft4\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366815    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091431d4-cb1e-46ef-babb-e5f54e589476-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366830    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366843    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366857    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366869    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366883    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366897    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091431d4-cb1e-46ef-babb-e5f54e589476-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.366939    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.367254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.367057    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istiod-ca-cert" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:30.368299 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.368271    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-token" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.368421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.368307    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-token" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.368743 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.368717    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-envoy" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.368855 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.368777    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-podinfo" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:30.369146 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.369123    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-podinfo" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:30.369276 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.369259    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-kube-api-access-h4rld" pod "c56eace7-1ab2-4646-9298-9f652a46bfff" (UID: "c56eace7-1ab2-4646-9298-9f652a46bfff"). InnerVolumeSpecName "kube-api-access-h4rld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.369521 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.369503    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-envoy" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.369932 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.369913    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-kube-api-access-z9bc7" pod "3d97a9b0-9aed-43ea-90ed-d280767bc9cc" (UID: "3d97a9b0-9aed-43ea-90ed-d280767bc9cc"). InnerVolumeSpecName "kube-api-access-z9bc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.468153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468049    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468092    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rld\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-kube-api-access-h4rld\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468110    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468125    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468135    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468143    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468151    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468153 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468159    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56eace7-1ab2-4646-9298-9f652a46bfff-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468515 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468171    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468515 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468179    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.468515 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.468187    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9bc7\" (UniqueName: \"kubernetes.io/projected/3d97a9b0-9aed-43ea-90ed-d280767bc9cc-kube-api-access-z9bc7\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.495309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.495277    4742 generic.go:356] "Generic (PLEG): container finished" podID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerID="794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330" exitCode=0
Oct 29 23:01:30.495468 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.495330    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"
Oct 29 23:01:30.495468 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.495357    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" event={"ID":"3d97a9b0-9aed-43ea-90ed-d280767bc9cc","Type":"ContainerDied","Data":"794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330"}
Oct 29 23:01:30.495468 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.495418    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t" event={"ID":"3d97a9b0-9aed-43ea-90ed-d280767bc9cc","Type":"ContainerDied","Data":"605dca7779426edd682d65f21471ed07ff254d41a529f3e29980e69a77718f86"}
Oct 29 23:01:30.495468 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.495453    4742 scope.go:122] "RemoveContainer" containerID="794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330"
Oct 29 23:01:30.497202 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.497173    4742 generic.go:356] "Generic (PLEG): container finished" podID="091431d4-cb1e-46ef-babb-e5f54e589476" containerID="70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4" exitCode=0
Oct 29 23:01:30.497305 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.497203    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" event={"ID":"091431d4-cb1e-46ef-babb-e5f54e589476","Type":"ContainerDied","Data":"70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4"}
Oct 29 23:01:30.497305 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.497226    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"
Oct 29 23:01:30.497305 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.497244    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67" event={"ID":"091431d4-cb1e-46ef-babb-e5f54e589476","Type":"ContainerDied","Data":"04d9e7bcdfcd776d751da11e780989ad3582650cb0dff73f73dd0b8f7ecec613"}
Oct 29 23:01:30.498780 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.498755    4742 generic.go:356] "Generic (PLEG): container finished" podID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerID="41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f" exitCode=0
Oct 29 23:01:30.498878 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.498802    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" event={"ID":"c56eace7-1ab2-4646-9298-9f652a46bfff","Type":"ContainerDied","Data":"41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f"}
Oct 29 23:01:30.498878 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.498837    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22" event={"ID":"c56eace7-1ab2-4646-9298-9f652a46bfff","Type":"ContainerDied","Data":"bacb188cd80b48463dbc9074d183b3bd3197af2209b549a8069e24e61403ad3f"}
Oct 29 23:01:30.498878 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.498805    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"
Oct 29 23:01:30.511324 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.511301    4742 scope.go:122] "RemoveContainer" containerID="794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330"
Oct 29 23:01:30.511673 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:30.511650    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330\": container with ID starting with 794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330 not found: ID does not exist" containerID="794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330"
Oct 29 23:01:30.511769 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.511682    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330"} err="failed to get container status \"794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330\": rpc error: code = NotFound desc = could not find container \"794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330\": container with ID starting with 794554ad3da73dcce1a6f410739aaeabb015919197b94aaf5b673397dd07a330 not found: ID does not exist"
Oct 29 23:01:30.511769 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.511702    4742 scope.go:122] "RemoveContainer" containerID="70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4"
Oct 29 23:01:30.526478 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.526440    4742 scope.go:122] "RemoveContainer" containerID="255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40"
Oct 29 23:01:30.536665 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.536637    4742 scope.go:122] "RemoveContainer" containerID="70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4"
Oct 29 23:01:30.536995 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:30.536977    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4\": container with ID starting with 70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4 not found: ID does not exist" containerID="70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4"
Oct 29 23:01:30.537057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.537006    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4"} err="failed to get container status \"70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4\": rpc error: code = NotFound desc = could not find container \"70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4\": container with ID starting with 70f08150ccb6fed9e48218342d891e5a07349e8298d36c49443f662b328ebec4 not found: ID does not exist"
Oct 29 23:01:30.537057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.537026    4742 scope.go:122] "RemoveContainer" containerID="255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40"
Oct 29 23:01:30.537352 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:30.537323    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40\": container with ID starting with 255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40 not found: ID does not exist" containerID="255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40"
Oct 29 23:01:30.537412 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.537363    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40"} err="failed to get container status \"255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40\": rpc error: code = NotFound desc = could not find container \"255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40\": container with ID starting with 255a30f0f00cff5a6c5ff25c4e7fb6f1c9a826fd17d0b7616e7b9d9ecf52af40 not found: ID does not exist"
Oct 29 23:01:30.537412 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.537385    4742 scope.go:122] "RemoveContainer" containerID="41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f"
Oct 29 23:01:30.546419 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.546396    4742 scope.go:122] "RemoveContainer" containerID="41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f"
Oct 29 23:01:30.546712 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:30.546691    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f\": container with ID starting with 41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f not found: ID does not exist" containerID="41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f"
Oct 29 23:01:30.546765 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.546721    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f"} err="failed to get container status \"41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f\": rpc error: code = NotFound desc = could not find container \"41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f\": container with ID starting with 41e0f54c8c44145592dba019d88456445b63585b25b93f08ac21103d1e62933f not found: ID does not exist"
Oct 29 23:01:30.557369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.557341    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"]
Oct 29 23:01:30.560209 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.560186    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-bzv67"]
Oct 29 23:01:30.592521 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.592485    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"]
Oct 29 23:01:30.597174 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.597133    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-5642t"]
Oct 29 23:01:30.628583 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.628551    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"]
Oct 29 23:01:30.632849 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.632819    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-44c22"]
Oct 29 23:01:30.991424 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.991384    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" path="/var/lib/kubelet/pods/091431d4-cb1e-46ef-babb-e5f54e589476/volumes"
Oct 29 23:01:30.991859 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.991844    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" path="/var/lib/kubelet/pods/3d97a9b0-9aed-43ea-90ed-d280767bc9cc/volumes"
Oct 29 23:01:30.992252 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:30.992239    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" path="/var/lib/kubelet/pods/c56eace7-1ab2-4646-9298-9f652a46bfff/volumes"
Oct 29 23:01:39.041739 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:39.041699    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"]
Oct 29 23:01:39.052775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:39.052742    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb90a3b-wb7d7"]
Oct 29 23:01:40.989078 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:40.989043    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9da0961-572b-49d6-80b4-76eb7e7aba38" path="/var/lib/kubelet/pods/d9da0961-572b-49d6-80b4-76eb7e7aba38/volumes"
Oct 29 23:01:52.037776 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:52.037728    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"]
Oct 29 23:01:52.050030 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:52.049987    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc849176-7b2cg"]
Oct 29 23:01:52.989206 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:52.989172    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d033e3-b179-46fd-9d28-25ebf951f150" path="/var/lib/kubelet/pods/c8d033e3-b179-46fd-9d28-25ebf951f150/volumes"
Oct 29 23:01:53.122213 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:01:53.122174    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:01:56.045337 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:56.045301    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"]
Oct 29 23:01:56.056692 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:56.056667    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b76cef9-w8zjs"]
Oct 29 23:01:56.989348 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:01:56.989313    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fa6f22-c750-43c0-ad37-49fe44224899" path="/var/lib/kubelet/pods/54fa6f22-c750-43c0-ad37-49fe44224899/volumes"
Oct 29 23:02:02.036539 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:02.036498    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"]
Oct 29 23:02:02.046839 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:02.046804    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d86588b-q7mng"]
Oct 29 23:02:02.995531 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:02.995496    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50163722-c649-46d9-911a-85ec7832e7f5" path="/var/lib/kubelet/pods/50163722-c649-46d9-911a-85ec7832e7f5/volumes"
Oct 29 23:02:08.037360 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:08.037312    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"]
Oct 29 23:02:08.047233 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:08.047206    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25aadaf-p8cb8"]
Oct 29 23:02:08.990369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:08.990335    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a827df17-86d8-4625-884f-d9eff3cdccdc" path="/var/lib/kubelet/pods/a827df17-86d8-4625-884f-d9eff3cdccdc/volumes"
Oct 29 23:02:14.032283 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.032251    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"]
Oct 29 23:02:14.041866 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.041840    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-975010f8-qfprf"]
Oct 29 23:02:14.331752 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.331661    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"]
Oct 29 23:02:14.332268 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332246    4742 cpu_manager.go:414] "containerMap: removing container" podUID="a08b4d2b-445e-4ae7-a976-419fc19ec87a" containerName="revoke-keys"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332270    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08b4d2b-445e-4ae7-a976-419fc19ec87a" containerName="revoke-keys"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332298    4742 cpu_manager.go:414] "containerMap: removing container" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332306    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332322    4742 cpu_manager.go:414] "containerMap: removing container" podUID="33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" containerName="payload-processing"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332330    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" containerName="payload-processing"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332341    4742 cpu_manager.go:414] "containerMap: removing container" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332349    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332360    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332368    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332377    4742 cpu_manager.go:414] "containerMap: removing container" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332384    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy"
Oct 29 23:02:14.332394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332394    4742 cpu_manager.go:414] "containerMap: removing container" podUID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" containerName="payload-pre-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332402    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" containerName="payload-pre-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332414    4742 cpu_manager.go:414] "containerMap: removing container" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332421    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332430    4742 cpu_manager.go:414] "containerMap: removing container" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332436    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332450    4742 cpu_manager.go:414] "containerMap: removing container" podUID="2e961f1d-5b31-4639-98a8-018c1060b806" containerName="payload-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332457    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e961f1d-5b31-4639-98a8-018c1060b806" containerName="payload-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332465    4742 cpu_manager.go:414] "containerMap: removing container" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332472    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332487    4742 cpu_manager.go:414] "containerMap: removing container" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332494    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332502    4742 cpu_manager.go:414] "containerMap: removing container" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332509    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332518    4742 cpu_manager.go:414] "containerMap: removing container" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332525    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332537    4742 cpu_manager.go:414] "containerMap: removing container" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332544    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332553    4742 cpu_manager.go:414] "containerMap: removing container" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332559    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332570    4742 cpu_manager.go:414] "containerMap: removing container" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332577    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332589    4742 cpu_manager.go:414] "containerMap: removing container" podUID="f952eacf-1713-4dba-890b-ac9838b1636b" containerName="payload-pre-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332596    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f952eacf-1713-4dba-890b-ac9838b1636b" containerName="payload-pre-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332607    4742 cpu_manager.go:414] "containerMap: removing container" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332614    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332629    4742 cpu_manager.go:414] "containerMap: removing container" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332637    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332646    4742 cpu_manager.go:414] "containerMap: removing container" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332653    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332664    4742 cpu_manager.go:414] "containerMap: removing container" podUID="42786f94-d2b2-493c-b722-2c0ef835e650" containerName="revoke-keys"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332671    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42786f94-d2b2-493c-b722-2c0ef835e650" containerName="revoke-keys"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332685    4742 cpu_manager.go:414] "containerMap: removing container" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332692    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332796    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332820    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="091431d4-cb1e-46ef-babb-e5f54e589476" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332830    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="42786f94-d2b2-493c-b722-2c0ef835e650" containerName="revoke-keys"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332841    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f0e924-1d9f-43a0-8a63-aaebf5ecc3d2" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332850    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08b4d2b-445e-4ae7-a976-419fc19ec87a" containerName="revoke-keys"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332861    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc9dad8-a255-4d7a-b627-b4ac73b8b615" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332869    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="33bcb0b6-9f48-4a6e-bd40-111de6a3cf23" containerName="payload-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332877    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="060de561-99f2-4378-b674-47d1e223a1c8" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332887    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332895    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f9d83d-a5a8-4739-9769-bdc9c275ab7a" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332904    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61d8c13-0e9f-4f30-be94-cddeb5abb510" containerName="payload-pre-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332912    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7fd15e-1f5c-407a-b86a-677851e1731b" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332919    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf42312-d61e-4b09-8a1b-37acd752ce5c" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332928    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e961f1d-5b31-4639-98a8-018c1060b806" containerName="payload-processing"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332935    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3eee93-3339-4aab-aa07-b7afcf5a56ca" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332944    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56eace7-1ab2-4646-9298-9f652a46bfff" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332951    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddcb392-8e28-4cce-984c-1f06108f927a" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332959    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4124490-af35-452c-8b90-2d878d194412" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332967    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332974    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="278c89c3-74ca-422d-b7c0-e13880664f59" containerName="main"
Oct 29 23:02:14.332962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332982    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d97a9b0-9aed-43ea-90ed-d280767bc9cc" containerName="istio-proxy"
Oct 29 23:02:14.335102 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332990    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="26438b13-81c6-4c88-a88e-19bceef48774" containerName="istio-proxy"
Oct 29 23:02:14.335102 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.332999    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="f952eacf-1713-4dba-890b-ac9838b1636b" containerName="payload-pre-processing"
Oct 29 23:02:14.335102 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.333008    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="692fbafb-57c8-4794-a6c4-c159075ae618" containerName="istio-proxy"
Oct 29 23:02:14.336441 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.336421    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.351421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.351399    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:02:14.351542 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.351484    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:02:14.362196 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.362166    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:02:14.362592 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.362571    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:02:14.366374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.366346    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"]
Oct 29 23:02:14.406308 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.406282    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.406468 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.406327    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfcf\" (UniqueName: \"kubernetes.io/projected/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-kube-api-access-zxfcf\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.406517 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.406466    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.507945 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.507912    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.508142 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.507958    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfcf\" (UniqueName: \"kubernetes.io/projected/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-kube-api-access-zxfcf\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.508142 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.508038    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.508619 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.508597    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.510455 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.510430    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.526718 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.526688    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfcf\" (UniqueName: \"kubernetes.io/projected/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-kube-api-access-zxfcf\") pod \"maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.649110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.649049    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:14.945772 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.945746    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"]
Oct 29 23:02:14.947006 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:02:14.946963    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb51fd49_bec1_45bb_9e8b_45d5ad8e7bba.slice/crio-9fc62e92af848a39031fa5a66695aef6747de96202e9dc78beabe85af6736331 WatchSource:0}: Error finding container 9fc62e92af848a39031fa5a66695aef6747de96202e9dc78beabe85af6736331: Status 404 returned error can't find the container with id 9fc62e92af848a39031fa5a66695aef6747de96202e9dc78beabe85af6736331
Oct 29 23:02:14.990016 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:14.989989    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c66fb512-f4b1-40dd-9650-d1c20240cbc4" path="/var/lib/kubelet/pods/c66fb512-f4b1-40dd-9650-d1c20240cbc4/volumes"
Oct 29 23:02:15.767923 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:15.767890    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv" event={"ID":"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba","Type":"ContainerStarted","Data":"a770fe1cf82f9e5d883afeb1d2421a57d576f9d1843619bca3a086141b0d3e51"}
Oct 29 23:02:15.768289 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:15.767954    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv" event={"ID":"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba","Type":"ContainerStarted","Data":"9fc62e92af848a39031fa5a66695aef6747de96202e9dc78beabe85af6736331"}
Oct 29 23:02:15.804668 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:15.804616    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv" podStartSLOduration=1.8046010049999999 podStartE2EDuration="1.804601005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:15.802644204 +0000 UTC m=+1192.943385801" watchObservedRunningTime="2026-10-29 23:02:15.804601005 +0000 UTC m=+1192.945342589"
Oct 29 23:02:16.775667 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:16.775632    4742 generic.go:356] "Generic (PLEG): container finished" podID="eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba" containerID="a770fe1cf82f9e5d883afeb1d2421a57d576f9d1843619bca3a086141b0d3e51" exitCode=0
Oct 29 23:02:16.776140 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:16.775721    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv" event={"ID":"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba","Type":"ContainerDied","Data":"a770fe1cf82f9e5d883afeb1d2421a57d576f9d1843619bca3a086141b0d3e51"}
Oct 29 23:02:17.915316 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:17.915291    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:18.045353 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.045257    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-kube-api-access-zxfcf\" (UniqueName: \"kubernetes.io/projected/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-kube-api-access-zxfcf\") pod \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") "
Oct 29 23:02:18.045353 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.045329    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-openshift-ca\") pod \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") "
Oct 29 23:02:18.045589 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.045441    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-xks-ca\") pod \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\" (UID: \"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba\") "
Oct 29 23:02:18.045719 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.045694    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-openshift-ca" pod "eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba" (UID: "eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:18.045831 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.045813    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.047381 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.047358    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-kube-api-access-zxfcf" pod "eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba" (UID: "eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba"). InnerVolumeSpecName "kube-api-access-zxfcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:18.047381 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.047365    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-xks-ca" pod "eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba" (UID: "eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:18.146955 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.146905    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.146955 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.146945    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxfcf\" (UniqueName: \"kubernetes.io/projected/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba-kube-api-access-zxfcf\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.788380 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.788351    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"
Oct 29 23:02:18.788380 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.788362    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv" event={"ID":"eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba","Type":"ContainerDied","Data":"9fc62e92af848a39031fa5a66695aef6747de96202e9dc78beabe85af6736331"}
Oct 29 23:02:18.788648 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:18.788432    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc62e92af848a39031fa5a66695aef6747de96202e9dc78beabe85af6736331"
Oct 29 23:02:19.115299 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:19.115266    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"]
Oct 29 23:02:19.144784 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:19.144750    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"]
Oct 29 23:02:20.033795 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:20.033757    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"]
Oct 29 23:02:20.043830 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:20.043799    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-179f2906-z5jch"]
Oct 29 23:02:20.989522 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:20.989486    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1" path="/var/lib/kubelet/pods/2cfc5f76-cc2c-4b8c-a60c-73717e53e7b1/volumes"
Oct 29 23:02:23.126782 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:02:23.126730    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:02:23.528999 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:23.528916    4742 scope.go:122] "RemoveContainer" containerID="290b21423a9e6505f52b16acb1e0bc1c49cdae1d84a720a6ae78956bb96ff1a4"
Oct 29 23:02:24.036548 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:24.036519    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"]
Oct 29 23:02:24.044433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:24.044406    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c09b16bf-z4t9q"]
Oct 29 23:02:24.991044 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:24.991014    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff524b22-dd9b-4916-8024-e6acb9c39697" path="/var/lib/kubelet/pods/ff524b22-dd9b-4916-8024-e6acb9c39697/volumes"
Oct 29 23:02:25.953591 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.953555    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"]
Oct 29 23:02:25.953940 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.953928    4742 cpu_manager.go:414] "containerMap: removing container" podUID="eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba" containerName="revoke-keys"
Oct 29 23:02:25.953985 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.953941    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba" containerName="revoke-keys"
Oct 29 23:02:25.953985 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.953954    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy"
Oct 29 23:02:25.953985 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.953959    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57f7435-1e6b-4e25-af1a-c399471ca25d" containerName="istio-proxy"
Oct 29 23:02:25.954077 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.954020    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba" containerName="revoke-keys"
Oct 29 23:02:25.958560 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.958543    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:25.978347 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.978315    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"]
Oct 29 23:02:25.982235 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.982208    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:25.985467 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.985444    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:02:25.986108 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.986064    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:02:25.986236 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.986215    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:02:25.986295 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.986273    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:02:25.987859 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:25.987834    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"]
Oct 29 23:02:26.001667 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.001640    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"]
Oct 29 23:02:26.125817 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.125772    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.126026 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.125837    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.126026 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.125924    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.126179 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.126042    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.126179 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.126076    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xr6\" (UniqueName: \"kubernetes.io/projected/64ed274b-9c66-4785-8a63-e3e47304148d-kube-api-access-z4xr6\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.126179 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.126146    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlnr\" (UniqueName: \"kubernetes.io/projected/f4a505dd-0531-43fe-8b46-1084006009c1-kube-api-access-lxlnr\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.227450 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.227342    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.227450 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.227395    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xr6\" (UniqueName: \"kubernetes.io/projected/64ed274b-9c66-4785-8a63-e3e47304148d-kube-api-access-z4xr6\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.227450 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.227441    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlnr\" (UniqueName: \"kubernetes.io/projected/f4a505dd-0531-43fe-8b46-1084006009c1-kube-api-access-lxlnr\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.227726 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.227585    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.227726 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.227648    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.227726 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.227680    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.228326 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.228302    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.228435 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.228303    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.229666 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.229638    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.229901 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.229881    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.244904 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.244878    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlnr\" (UniqueName: \"kubernetes.io/projected/f4a505dd-0531-43fe-8b46-1084006009c1-kube-api-access-lxlnr\") pod \"maas-api-revoke-sub-0cde6278-v46ct\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.244904 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.244892    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xr6\" (UniqueName: \"kubernetes.io/projected/64ed274b-9c66-4785-8a63-e3e47304148d-kube-api-access-z4xr6\") pod \"maas-api-revoke-sub-37ecc37f-vcd4m\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.271478 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.271448    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:26.295563 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.295534    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:26.571522 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.571488    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"]
Oct 29 23:02:26.572235 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:02:26.572210    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a505dd_0531_43fe_8b46_1084006009c1.slice/crio-891ba4ebcee81dc9ea6df4f719e0443e12d2ab130a56e5c39fb5be71669f484e WatchSource:0}: Error finding container 891ba4ebcee81dc9ea6df4f719e0443e12d2ab130a56e5c39fb5be71669f484e: Status 404 returned error can't find the container with id 891ba4ebcee81dc9ea6df4f719e0443e12d2ab130a56e5c39fb5be71669f484e
Oct 29 23:02:26.600696 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.600667    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"]
Oct 29 23:02:26.601882 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:02:26.601853    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ed274b_9c66_4785_8a63_e3e47304148d.slice/crio-52fd0a907f075a67abff8aeeed5bd862131126de9aaaba1a376301ac7407f106 WatchSource:0}: Error finding container 52fd0a907f075a67abff8aeeed5bd862131126de9aaaba1a376301ac7407f106: Status 404 returned error can't find the container with id 52fd0a907f075a67abff8aeeed5bd862131126de9aaaba1a376301ac7407f106
Oct 29 23:02:26.837838 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.837808    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m" event={"ID":"64ed274b-9c66-4785-8a63-e3e47304148d","Type":"ContainerStarted","Data":"52fd0a907f075a67abff8aeeed5bd862131126de9aaaba1a376301ac7407f106"}
Oct 29 23:02:26.839138 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:26.839108    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct" event={"ID":"f4a505dd-0531-43fe-8b46-1084006009c1","Type":"ContainerStarted","Data":"891ba4ebcee81dc9ea6df4f719e0443e12d2ab130a56e5c39fb5be71669f484e"}
Oct 29 23:02:27.027482 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:27.027446    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"]
Oct 29 23:02:27.031828 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:27.031799    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba7ef576-p94bm"]
Oct 29 23:02:27.847353 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:27.847316    4742 generic.go:356] "Generic (PLEG): container finished" podID="64ed274b-9c66-4785-8a63-e3e47304148d" containerID="39d570d8f251de25f1c06e586631a6e147fad37e61b2b926d7be8469d8f44ebe" exitCode=0
Oct 29 23:02:27.847566 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:27.847408    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m" event={"ID":"64ed274b-9c66-4785-8a63-e3e47304148d","Type":"ContainerDied","Data":"39d570d8f251de25f1c06e586631a6e147fad37e61b2b926d7be8469d8f44ebe"}
Oct 29 23:02:27.849057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:27.849030    4742 generic.go:356] "Generic (PLEG): container finished" podID="f4a505dd-0531-43fe-8b46-1084006009c1" containerID="7b4b382372f34cb2f7c404b389d5ac2baf8e141ad6fed4dd5bc5eea04e6910c5" exitCode=0
Oct 29 23:02:27.849195 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:27.849111    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct" event={"ID":"f4a505dd-0531-43fe-8b46-1084006009c1","Type":"ContainerDied","Data":"7b4b382372f34cb2f7c404b389d5ac2baf8e141ad6fed4dd5bc5eea04e6910c5"}
Oct 29 23:02:28.991003 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:28.990973    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06313a9-f24b-4c7d-aaa3-79351596a292" path="/var/lib/kubelet/pods/a06313a9-f24b-4c7d-aaa3-79351596a292/volumes"
Oct 29 23:02:29.010020 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.009993    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:29.012658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.012639    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:29.059398 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.059366    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-openshift-ca\") pod \"f4a505dd-0531-43fe-8b46-1084006009c1\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") "
Oct 29 23:02:29.059547 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.059404    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-xks-ca\") pod \"64ed274b-9c66-4785-8a63-e3e47304148d\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") "
Oct 29 23:02:29.059547 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.059437    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-openshift-ca\") pod \"64ed274b-9c66-4785-8a63-e3e47304148d\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") "
Oct 29 23:02:29.059547 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.059499    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ed274b-9c66-4785-8a63-e3e47304148d-kube-api-access-z4xr6\" (UniqueName: \"kubernetes.io/projected/64ed274b-9c66-4785-8a63-e3e47304148d-kube-api-access-z4xr6\") pod \"64ed274b-9c66-4785-8a63-e3e47304148d\" (UID: \"64ed274b-9c66-4785-8a63-e3e47304148d\") "
Oct 29 23:02:29.059547 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.059528    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-xks-ca\") pod \"f4a505dd-0531-43fe-8b46-1084006009c1\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") "
Oct 29 23:02:29.059774 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.059601    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a505dd-0531-43fe-8b46-1084006009c1-kube-api-access-lxlnr\" (UniqueName: \"kubernetes.io/projected/f4a505dd-0531-43fe-8b46-1084006009c1-kube-api-access-lxlnr\") pod \"f4a505dd-0531-43fe-8b46-1084006009c1\" (UID: \"f4a505dd-0531-43fe-8b46-1084006009c1\") "
Oct 29 23:02:29.059835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.059807    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-openshift-ca" pod "f4a505dd-0531-43fe-8b46-1084006009c1" (UID: "f4a505dd-0531-43fe-8b46-1084006009c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:29.059890 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.059855    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-openshift-ca" pod "64ed274b-9c66-4785-8a63-e3e47304148d" (UID: "64ed274b-9c66-4785-8a63-e3e47304148d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:29.060060 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.060038    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.060176 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.060062    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.061826 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.061800    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ed274b-9c66-4785-8a63-e3e47304148d-kube-api-access-z4xr6" pod "64ed274b-9c66-4785-8a63-e3e47304148d" (UID: "64ed274b-9c66-4785-8a63-e3e47304148d"). InnerVolumeSpecName "kube-api-access-z4xr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:29.061972 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.061928    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-xks-ca" pod "f4a505dd-0531-43fe-8b46-1084006009c1" (UID: "f4a505dd-0531-43fe-8b46-1084006009c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:29.062139 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.062111    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-xks-ca" pod "64ed274b-9c66-4785-8a63-e3e47304148d" (UID: "64ed274b-9c66-4785-8a63-e3e47304148d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:29.062139 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.062123    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a505dd-0531-43fe-8b46-1084006009c1-kube-api-access-lxlnr" pod "f4a505dd-0531-43fe-8b46-1084006009c1" (UID: "f4a505dd-0531-43fe-8b46-1084006009c1"). InnerVolumeSpecName "kube-api-access-lxlnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:29.160717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.160622    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlnr\" (UniqueName: \"kubernetes.io/projected/f4a505dd-0531-43fe-8b46-1084006009c1-kube-api-access-lxlnr\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.160717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.160656    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ed274b-9c66-4785-8a63-e3e47304148d-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.160717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.160666    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4xr6\" (UniqueName: \"kubernetes.io/projected/64ed274b-9c66-4785-8a63-e3e47304148d-kube-api-access-z4xr6\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.160717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.160676    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a505dd-0531-43fe-8b46-1084006009c1-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.869265 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.869232    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"
Oct 29 23:02:29.869470 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.869231    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m" event={"ID":"64ed274b-9c66-4785-8a63-e3e47304148d","Type":"ContainerDied","Data":"52fd0a907f075a67abff8aeeed5bd862131126de9aaaba1a376301ac7407f106"}
Oct 29 23:02:29.869470 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.869397    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52fd0a907f075a67abff8aeeed5bd862131126de9aaaba1a376301ac7407f106"
Oct 29 23:02:29.870740 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.870723    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"
Oct 29 23:02:29.870854 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.870757    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct" event={"ID":"f4a505dd-0531-43fe-8b46-1084006009c1","Type":"ContainerDied","Data":"891ba4ebcee81dc9ea6df4f719e0443e12d2ab130a56e5c39fb5be71669f484e"}
Oct 29 23:02:29.870854 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:29.870789    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="891ba4ebcee81dc9ea6df4f719e0443e12d2ab130a56e5c39fb5be71669f484e"
Oct 29 23:02:34.134831 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.134771    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main" containerID="cri-o://cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39" gracePeriod=45
Oct 29 23:02:34.163961 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.163924    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main" containerID="cri-o://aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be" gracePeriod=45
Oct 29 23:02:34.422815 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.422789    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 23:02:34.457445 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.457417    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 23:02:34.512097 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512044    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-dshm\") pod \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") "
Oct 29 23:02:34.512294 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512121    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tmp-dir\") pod \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") "
Oct 29 23:02:34.512294 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512167    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-kube-api-access-c2pmf\" (UniqueName: \"kubernetes.io/projected/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-kube-api-access-c2pmf\") pod \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") "
Oct 29 23:02:34.512294 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512226    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-home\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-home\") pod \"920def85-325f-4d23-b508-2b036320e3bb\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") "
Oct 29 23:02:34.512294 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512266    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/920def85-325f-4d23-b508-2b036320e3bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/920def85-325f-4d23-b508-2b036320e3bb-tls-certs\") pod \"920def85-325f-4d23-b508-2b036320e3bb\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") "
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512310    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-model-cache\") pod \"920def85-325f-4d23-b508-2b036320e3bb\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") "
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512345    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-tmp-dir\") pod \"920def85-325f-4d23-b508-2b036320e3bb\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") "
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512404    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-model-cache\") pod \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") "
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512410    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tmp-dir" pod "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" (UID: "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512428    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-home\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-home\") pod \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") "
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512460    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-dshm\") pod \"920def85-325f-4d23-b508-2b036320e3bb\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") "
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512489    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tls-certs\") pod \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\" (UID: \"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02\") "
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512632    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-model-cache" pod "920def85-325f-4d23-b508-2b036320e3bb" (UID: "920def85-325f-4d23-b508-2b036320e3bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512708    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920def85-325f-4d23-b508-2b036320e3bb-kube-api-access-fcgld\" (UniqueName: \"kubernetes.io/projected/920def85-325f-4d23-b508-2b036320e3bb-kube-api-access-fcgld\") pod \"920def85-325f-4d23-b508-2b036320e3bb\" (UID: \"920def85-325f-4d23-b508-2b036320e3bb\") "
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.513172    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.513194    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512712    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-home" pod "920def85-325f-4d23-b508-2b036320e3bb" (UID: "920def85-325f-4d23-b508-2b036320e3bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.512876    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-tmp-dir" pod "920def85-325f-4d23-b508-2b036320e3bb" (UID: "920def85-325f-4d23-b508-2b036320e3bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.513108    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-model-cache" pod "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" (UID: "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:34.513642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.513306    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-home" pod "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" (UID: "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:34.515371 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.515344    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-kube-api-access-c2pmf" pod "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" (UID: "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02"). InnerVolumeSpecName "kube-api-access-c2pmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:34.515371 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.515357    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920def85-325f-4d23-b508-2b036320e3bb-kube-api-access-fcgld" pod "920def85-325f-4d23-b508-2b036320e3bb" (UID: "920def85-325f-4d23-b508-2b036320e3bb"). InnerVolumeSpecName "kube-api-access-fcgld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:34.515529 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.515405    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-dshm" pod "920def85-325f-4d23-b508-2b036320e3bb" (UID: "920def85-325f-4d23-b508-2b036320e3bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:34.515529 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.515435    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920def85-325f-4d23-b508-2b036320e3bb-tls-certs" pod "920def85-325f-4d23-b508-2b036320e3bb" (UID: "920def85-325f-4d23-b508-2b036320e3bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:34.516507 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.516475    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-dshm" pod "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" (UID: "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:34.517681 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.517660    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tls-certs" pod "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" (UID: "d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:34.614171 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614124    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614192    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2pmf\" (UniqueName: \"kubernetes.io/projected/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-kube-api-access-c2pmf\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614208    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614221    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920def85-325f-4d23-b508-2b036320e3bb-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614233    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614245    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614256    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614266    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920def85-325f-4d23-b508-2b036320e3bb-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614276    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.614369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.614287    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcgld\" (UniqueName: \"kubernetes.io/projected/920def85-325f-4d23-b508-2b036320e3bb-kube-api-access-fcgld\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:34.901717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.901674    4742 generic.go:356] "Generic (PLEG): container finished" podID="920def85-325f-4d23-b508-2b036320e3bb" containerID="aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be" exitCode=0
Oct 29 23:02:34.901927 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.901729    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"
Oct 29 23:02:34.901927 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.901726    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" event={"ID":"920def85-325f-4d23-b508-2b036320e3bb","Type":"ContainerDied","Data":"aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be"}
Oct 29 23:02:34.901927 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.901791    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2" event={"ID":"920def85-325f-4d23-b508-2b036320e3bb","Type":"ContainerDied","Data":"bee3cdbc518d0760a24c5a931b4845c56b7e768d45a9377913c1b797acfa40a6"}
Oct 29 23:02:34.901927 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.901815    4742 scope.go:122] "RemoveContainer" containerID="aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be"
Oct 29 23:02:34.903264 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.903233    4742 generic.go:356] "Generic (PLEG): container finished" podID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerID="cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39" exitCode=0
Oct 29 23:02:34.903370 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.903276    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" event={"ID":"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02","Type":"ContainerDied","Data":"cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39"}
Oct 29 23:02:34.903370 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.903292    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"
Oct 29 23:02:34.903370 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.903311    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz" event={"ID":"d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02","Type":"ContainerDied","Data":"29ffcca9eff26db3e8808bd8f0b1d68b2eb40ff6a3274e9bceca8810dcedbec3"}
Oct 29 23:02:34.911822 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.911797    4742 scope.go:122] "RemoveContainer" containerID="aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be"
Oct 29 23:02:34.912205 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:02:34.912181    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be\": container with ID starting with aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be not found: ID does not exist" containerID="aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be"
Oct 29 23:02:34.912294 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.912214    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be"} err="failed to get container status \"aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be\": rpc error: code = NotFound desc = could not find container \"aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be\": container with ID starting with aa80bd3e8999e61babdb9f3212546e7cbd012fda81319c377dc0db9f992d40be not found: ID does not exist"
Oct 29 23:02:34.912294 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.912238    4742 scope.go:122] "RemoveContainer" containerID="cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39"
Oct 29 23:02:34.920919 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.920901    4742 scope.go:122] "RemoveContainer" containerID="cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39"
Oct 29 23:02:34.921258 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:02:34.921238    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39\": container with ID starting with cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39 not found: ID does not exist" containerID="cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39"
Oct 29 23:02:34.921313 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.921268    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39"} err="failed to get container status \"cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39\": rpc error: code = NotFound desc = could not find container \"cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39\": container with ID starting with cc7b1103a0f73a1c66ed7635e0928d91256107f56bf3192425500f01504e7d39 not found: ID does not exist"
Oct 29 23:02:34.950862 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.950819    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"]
Oct 29 23:02:34.959404 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.959373    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/distinct-2-w0-ffba57-kserve-69f64ff8c5-tnsjz"]
Oct 29 23:02:34.990073 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.990037    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" path="/var/lib/kubelet/pods/d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02/volumes"
Oct 29 23:02:34.993388 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.993349    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"]
Oct 29 23:02:34.995763 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:34.995736    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ffba57/simulator-w0-ffba57-kserve-5d4fc69db8-dhhn2"]
Oct 29 23:02:35.034277 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:35.034232    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"]
Oct 29 23:02:35.037221 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:35.037193    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2024ac24-p8npk"]
Oct 29 23:02:36.990575 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:36.990501    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920def85-325f-4d23-b508-2b036320e3bb" path="/var/lib/kubelet/pods/920def85-325f-4d23-b508-2b036320e3bb/volumes"
Oct 29 23:02:36.990976 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:36.990959    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05c7896-82c6-4b1a-ac33-339d90924c67" path="/var/lib/kubelet/pods/a05c7896-82c6-4b1a-ac33-339d90924c67/volumes"
Oct 29 23:02:39.026744 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:39.026708    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"]
Oct 29 23:02:39.040165 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:39.040120    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9c9aa3-gck8h"]
Oct 29 23:02:40.033733 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:40.033701    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"]
Oct 29 23:02:40.043620 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:40.043586    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33426e78-8wxw2"]
Oct 29 23:02:40.989602 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:40.989567    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ded313-855e-471d-b90a-80bb77379d21" path="/var/lib/kubelet/pods/a6ded313-855e-471d-b90a-80bb77379d21/volumes"
Oct 29 23:02:40.989989 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:40.989976    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f4d451-021f-4db2-838d-1aae8ebeddca" path="/var/lib/kubelet/pods/b7f4d451-021f-4db2-838d-1aae8ebeddca/volumes"
Oct 29 23:02:41.804363 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:41.804330    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"]
Oct 29 23:02:41.816349 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:41.816315    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ffba57-hvnxv"]
Oct 29 23:02:42.989998 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:42.989967    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba" path="/var/lib/kubelet/pods/eb51fd49-bec1-45bb-9e8b-45d5ad8e7bba/volumes"
Oct 29 23:02:50.029386 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:50.029346    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"]
Oct 29 23:02:50.029921 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:50.029677    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" containerID="cri-o://0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd" gracePeriod=30
Oct 29 23:02:50.038949 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:50.038921    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"]
Oct 29 23:02:50.039205 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:50.039176    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy" containerID="cri-o://af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02" gracePeriod=30
Oct 29 23:02:50.041625 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:50.041601    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"]
Oct 29 23:02:50.041829 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:50.041802    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" containerID="cri-o://5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70" gracePeriod=30
Oct 29 23:02:50.313788 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:50.313708    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:02:50.313788 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:50.313767    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:02:52.024494 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:52.024457    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"]
Oct 29 23:02:52.034902 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:52.034872    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25637484-nhkhv"]
Oct 29 23:02:52.989451 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:52.989417    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e5badd-a4a2-43fe-bbab-a1d5249ad4a5" path="/var/lib/kubelet/pods/92e5badd-a4a2-43fe-bbab-a1d5249ad4a5/volumes"
Oct 29 23:02:53.123320 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:02:53.123279    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:02:53.751135 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:53.751100    4742 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:02:53.751325 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:53.751169    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:02:54.100286 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100258    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-848f956548-9fnqq"]
Oct 29 23:02:54.100612 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100600    4742 cpu_manager.go:414] "containerMap: removing container" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main"
Oct 29 23:02:54.100658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100613    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main"
Oct 29 23:02:54.100658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100634    4742 cpu_manager.go:414] "containerMap: removing container" podUID="f4a505dd-0531-43fe-8b46-1084006009c1" containerName="revoke-keys"
Oct 29 23:02:54.100658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100639    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a505dd-0531-43fe-8b46-1084006009c1" containerName="revoke-keys"
Oct 29 23:02:54.100658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100654    4742 cpu_manager.go:414] "containerMap: removing container" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main"
Oct 29 23:02:54.100658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100658    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main"
Oct 29 23:02:54.100813 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100671    4742 cpu_manager.go:414] "containerMap: removing container" podUID="64ed274b-9c66-4785-8a63-e3e47304148d" containerName="revoke-keys"
Oct 29 23:02:54.100813 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100676    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ed274b-9c66-4785-8a63-e3e47304148d" containerName="revoke-keys"
Oct 29 23:02:54.100813 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100727    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b6fd0a-d942-4ed0-9f95-19dc80ff7b02" containerName="main"
Oct 29 23:02:54.100813 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100737    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ed274b-9c66-4785-8a63-e3e47304148d" containerName="revoke-keys"
Oct 29 23:02:54.100813 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100743    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="920def85-325f-4d23-b508-2b036320e3bb" containerName="main"
Oct 29 23:02:54.100813 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.100751    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a505dd-0531-43fe-8b46-1084006009c1" containerName="revoke-keys"
Oct 29 23:02:54.104969 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.104949    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.119708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.119683    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 29 23:02:54.130070 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.130044    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jmzz4\""
Oct 29 23:02:54.130070 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.130044    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 29 23:02:54.130470 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.130155    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 29 23:02:54.132939 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.132916    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848f956548-9fnqq"]
Oct 29 23:02:54.246740 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.246707    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjt9\" (UniqueName: \"kubernetes.io/projected/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-kube-api-access-nhjt9\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.246903 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.246758    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-webhook-cert\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.246903 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.246859    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-metrics-certs\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.347380 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.347349    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjt9\" (UniqueName: \"kubernetes.io/projected/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-kube-api-access-nhjt9\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.347580 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.347415    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-webhook-cert\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.347580 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.347468    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-metrics-certs\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.349874 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.349847    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-webhook-cert\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.349979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.349876    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-metrics-certs\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.370982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.370918    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjt9\" (UniqueName: \"kubernetes.io/projected/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-kube-api-access-nhjt9\") pod \"maas-controller-848f956548-9fnqq\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") " pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.418204 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.418170    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:54.714606 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:54.714575    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848f956548-9fnqq"]
Oct 29 23:02:54.715677 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:02:54.715648    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0aa34ce_05b1_4a1d_8280_4213c725b1e8.slice/crio-04591a216a36f03582ccdfbfafd3a46d7bd87cc4345328ff91a34196dc83dd1f WatchSource:0}: Error finding container 04591a216a36f03582ccdfbfafd3a46d7bd87cc4345328ff91a34196dc83dd1f: Status 404 returned error can't find the container with id 04591a216a36f03582ccdfbfafd3a46d7bd87cc4345328ff91a34196dc83dd1f
Oct 29 23:02:55.031580 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.031488    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848f956548-9fnqq" event={"ID":"c0aa34ce-05b1-4a1d-8280-4213c725b1e8","Type":"ContainerStarted","Data":"04591a216a36f03582ccdfbfafd3a46d7bd87cc4345328ff91a34196dc83dd1f"}
Oct 29 23:02:55.287937 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.287868    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 23:02:55.328675 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.328652    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 23:02:55.331946 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.331929    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 23:02:55.461571 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461532    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-envoy\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.461779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461587    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-certs\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.461779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461628    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-certs\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.461779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461666    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-istio-token\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-istio-token\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.461779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461697    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-envoy\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.461779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461727    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef9b6036-f938-43b2-8519-184c45185696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9b6036-f938-43b2-8519-184c45185696-istio-podinfo\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.461779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461767    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-credential-socket\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.462127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461800    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-socket\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.462127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461843    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-socket\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.462127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461888    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-data\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.462127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461945    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-data\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.462127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.461980    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-credential-socket\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.462127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.462009    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istiod-ca-cert\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.462127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.462046    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-certs\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.462127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.462099    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istiod-ca-cert\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.462653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.462136    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-kube-api-access-b85fs\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-kube-api-access-b85fs\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.462653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.462177    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-socket\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.462653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.462212    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-podinfo\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.462653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.462251    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-podinfo\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.462653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.462294    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-envoy\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.463879 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.463768    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-credential-socket" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.464068 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.463974    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-certs" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.464514 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.464455    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-data" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.468654 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.464645    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istiod-ca-cert" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.468654 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.464477    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istiod-ca-cert" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.468654 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.465135    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-certs" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.468907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.465465    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-certs" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.468907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.467914    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-socket" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.468907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.468327    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-socket" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.468907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.468590    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-credential-socket" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.468907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.468734    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-data" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.468907 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.468782    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-socket" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.469392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.468926    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9b6036-f938-43b2-8519-184c45185696-istiod-ca-cert" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.469392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469022    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9b6036-f938-43b2-8519-184c45185696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9b6036-f938-43b2-8519-184c45185696-istiod-ca-cert\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.469392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469064    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-token\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.469392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469111    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-credential-socket\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.469392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469153    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-kube-api-access-qkg65\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-kube-api-access-qkg65\") pod \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\" (UID: \"7c4a636e-a3b0-4347-8b47-68c6b6dca509\") "
Oct 29 23:02:55.469392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469177    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-kube-api-access-r2mh4\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-kube-api-access-r2mh4\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.469392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469205    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-token\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-token\") pod \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\" (UID: \"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a\") "
Oct 29 23:02:55.469392 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469233    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-data\") pod \"ef9b6036-f938-43b2-8519-184c45185696\" (UID: \"ef9b6036-f938-43b2-8519-184c45185696\") "
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469565    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469580    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469594    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469606    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469617    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469627    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469638    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469652    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469664    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469674    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469685    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9b6036-f938-43b2-8519-184c45185696-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469696    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.469800 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.469707    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.485590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.473022    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-data" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.485590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.473928    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-credential-socket" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.485590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.478107    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-token" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.485590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.481272    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-kube-api-access-r2mh4" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "kube-api-access-r2mh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.485590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.481346    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef9b6036-f938-43b2-8519-184c45185696-istio-podinfo" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:55.485590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.481402    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-istio-token" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.485590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.481520    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-envoy" pod "ef9b6036-f938-43b2-8519-184c45185696" (UID: "ef9b6036-f938-43b2-8519-184c45185696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.488778 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.488617    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-kube-api-access-qkg65" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "kube-api-access-qkg65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.516336 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.500300    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-token" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.517638 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.517579    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-envoy" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.517638 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.517583    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-kube-api-access-b85fs" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "kube-api-access-b85fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.517988 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.517951    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-podinfo" pod "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" (UID: "9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:55.518145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.517957    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-envoy" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.519804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.519779    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-podinfo" pod "7c4a636e-a3b0-4347-8b47-68c6b6dca509" (UID: "7c4a636e-a3b0-4347-8b47-68c6b6dca509"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570484    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570517    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570527    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570538    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570547    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9b6036-f938-43b2-8519-184c45185696-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570555    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b85fs\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-kube-api-access-b85fs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570564    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570574    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.570577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570582    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6036-f938-43b2-8519-184c45185696-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.571281 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570591    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.571281 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570599    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.571281 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570607    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkg65\" (UniqueName: \"kubernetes.io/projected/7c4a636e-a3b0-4347-8b47-68c6b6dca509-kube-api-access-qkg65\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.571281 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570615    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2mh4\" (UniqueName: \"kubernetes.io/projected/ef9b6036-f938-43b2-8519-184c45185696-kube-api-access-r2mh4\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.571281 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:55.570623    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:02:56.053356 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.053314    4742 generic.go:356] "Generic (PLEG): container finished" podID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerID="0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd" exitCode=0
Oct 29 23:02:56.053584 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.053493    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" event={"ID":"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a","Type":"ContainerDied","Data":"0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd"}
Oct 29 23:02:56.053584 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.053549    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8" event={"ID":"9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a","Type":"ContainerDied","Data":"d80df30bcdc7919d532a43dd3620864046480af13eccd1f3714bb1a17eafd0fc"}
Oct 29 23:02:56.053584 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.053578    4742 scope.go:122] "RemoveContainer" containerID="0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd"
Oct 29 23:02:56.053783 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.053740    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"
Oct 29 23:02:56.078491 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.078457    4742 generic.go:356] "Generic (PLEG): container finished" podID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerID="af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02" exitCode=0
Oct 29 23:02:56.078883 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.078564    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"
Oct 29 23:02:56.079179 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.078589    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" event={"ID":"7c4a636e-a3b0-4347-8b47-68c6b6dca509","Type":"ContainerDied","Data":"af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02"}
Oct 29 23:02:56.079179 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.079009    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf" event={"ID":"7c4a636e-a3b0-4347-8b47-68c6b6dca509","Type":"ContainerDied","Data":"2a4a776f1cded10f1ff214a0df4a7c6684109629094fa2607466eb81e39a05b4"}
Oct 29 23:02:56.083145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.082960    4742 generic.go:356] "Generic (PLEG): container finished" podID="ef9b6036-f938-43b2-8519-184c45185696" containerID="5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70" exitCode=0
Oct 29 23:02:56.083145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.083009    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"
Oct 29 23:02:56.083145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.083019    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" event={"ID":"ef9b6036-f938-43b2-8519-184c45185696","Type":"ContainerDied","Data":"5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70"}
Oct 29 23:02:56.083145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.083055    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6" event={"ID":"ef9b6036-f938-43b2-8519-184c45185696","Type":"ContainerDied","Data":"2dc3b77b5d27d5092cbd5a46c14f4fe6bed59b844862088c2708ca29be94867c"}
Oct 29 23:02:56.086051 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.086026    4742 scope.go:122] "RemoveContainer" containerID="0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd"
Oct 29 23:02:56.086385 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:02:56.086357    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd\": container with ID starting with 0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd not found: ID does not exist" containerID="0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd"
Oct 29 23:02:56.086478 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.086397    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd"} err="failed to get container status \"0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd\": rpc error: code = NotFound desc = could not find container \"0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd\": container with ID starting with 0fcd0e737f2171e5bd2e63b102f1ed5124f8a4f9af1bfbd81e330e0936c979bd not found: ID does not exist"
Oct 29 23:02:56.086478 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.086420    4742 scope.go:122] "RemoveContainer" containerID="af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02"
Oct 29 23:02:56.106797 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.106772    4742 scope.go:122] "RemoveContainer" containerID="e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294"
Oct 29 23:02:56.120216 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.120175    4742 scope.go:122] "RemoveContainer" containerID="af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02"
Oct 29 23:02:56.120703 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:02:56.120671    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02\": container with ID starting with af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02 not found: ID does not exist" containerID="af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02"
Oct 29 23:02:56.120803 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.120717    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02"} err="failed to get container status \"af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02\": rpc error: code = NotFound desc = could not find container \"af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02\": container with ID starting with af5e7469951cd413e0d182de635124ce92e872b8950436e7420a7de504059b02 not found: ID does not exist"
Oct 29 23:02:56.120803 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.120747    4742 scope.go:122] "RemoveContainer" containerID="e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294"
Oct 29 23:02:56.121241 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:02:56.121218    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294\": container with ID starting with e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294 not found: ID does not exist" containerID="e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294"
Oct 29 23:02:56.121351 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.121246    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294"} err="failed to get container status \"e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294\": rpc error: code = NotFound desc = could not find container \"e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294\": container with ID starting with e0eb12c835041a58d9f4824f124df1b6a3462454831e525cfb75f66cb19d3294 not found: ID does not exist"
Oct 29 23:02:56.121351 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.121261    4742 scope.go:122] "RemoveContainer" containerID="5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70"
Oct 29 23:02:56.132887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.132859    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"]
Oct 29 23:02:56.136139 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.136109    4742 scope.go:122] "RemoveContainer" containerID="5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70"
Oct 29 23:02:56.136616 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:02:56.136550    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70\": container with ID starting with 5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70 not found: ID does not exist" containerID="5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70"
Oct 29 23:02:56.136616 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.136586    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70"} err="failed to get container status \"5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70\": rpc error: code = NotFound desc = could not find container \"5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70\": container with ID starting with 5ef77095e4c7a529b3c5115960e531b67a173c9036a85d98d55787d7c316fc70 not found: ID does not exist"
Oct 29 23:02:56.143866 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.143840    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-8swq8"]
Oct 29 23:02:56.181397 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.181098    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"]
Oct 29 23:02:56.184674 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.184644    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-5ddwf"]
Oct 29 23:02:56.213180 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.212548    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"]
Oct 29 23:02:56.219050 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:56.219016    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-f6jb6"]
Oct 29 23:02:57.010109 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:57.009103    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" path="/var/lib/kubelet/pods/7c4a636e-a3b0-4347-8b47-68c6b6dca509/volumes"
Oct 29 23:02:57.010109 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:57.009731    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" path="/var/lib/kubelet/pods/9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a/volumes"
Oct 29 23:02:57.010639 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:57.010319    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9b6036-f938-43b2-8519-184c45185696" path="/var/lib/kubelet/pods/ef9b6036-f938-43b2-8519-184c45185696/volumes"
Oct 29 23:02:57.138577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:57.138538    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848f956548-9fnqq" event={"ID":"c0aa34ce-05b1-4a1d-8280-4213c725b1e8","Type":"ContainerStarted","Data":"958757d7eb7f7c4fa255058dfb0c4657b61fbe8673d7faf28332964523a06121"}
Oct 29 23:02:57.138870 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:57.138853    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:02:57.222407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:02:57.222278    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-848f956548-9fnqq" podStartSLOduration=1.01525588 podStartE2EDuration="3.222249708s" totalImagesPullingTime="2.206993828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:57.181523627 +0000 UTC m=+1234.322265250" watchObservedRunningTime="2026-10-29 23:02:57.222249708 +0000 UTC m=+1234.362991293"
Oct 29 23:03:07.026165 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:07.026126    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"]
Oct 29 23:03:07.036414 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:07.036385    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a8622e1-j6sfp"]
Oct 29 23:03:08.151228 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:08.151201    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:03:08.989663 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:08.989632    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0fc532-f498-4b81-9be4-8d0ba34d3f9d" path="/var/lib/kubelet/pods/1b0fc532-f498-4b81-9be4-8d0ba34d3f9d/volumes"
Oct 29 23:03:11.027138 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:11.027097    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"]
Oct 29 23:03:11.037620 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:11.037581    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac3c53e-dxmxm"]
Oct 29 23:03:12.989804 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:12.989770    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3ec66b-f795-49b6-8b9a-fe0a43168784" path="/var/lib/kubelet/pods/1d3ec66b-f795-49b6-8b9a-fe0a43168784/volumes"
Oct 29 23:03:21.032936 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:21.032899    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"]
Oct 29 23:03:21.043582 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:21.043555    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987324d0-7jcd6"]
Oct 29 23:03:22.990170 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:22.990136    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="671014f5-a911-41e0-927f-200bdcb80a43" path="/var/lib/kubelet/pods/671014f5-a911-41e0-927f-200bdcb80a43/volumes"
Oct 29 23:03:23.122403 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:03:23.122362    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:03:23.568388 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:23.568352    4742 scope.go:122] "RemoveContainer" containerID="27864d3d035a58f9e7bddb4592ea25d4d20460a9c1e05502b96885c2982563bb"
Oct 29 23:03:23.577095 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:23.577067    4742 scope.go:122] "RemoveContainer" containerID="639ff5ec8e83f6dcd97fcd0e74ac4d7b3f74a9f78f2ea845335ac312509bbfed"
Oct 29 23:03:23.585384 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:23.585354    4742 scope.go:122] "RemoveContainer" containerID="d9a4f369202bd537f6ba03b51e5e13af306d4cba38e0bcd95ef6cfd8119e7fc7"
Oct 29 23:03:23.593059 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:23.593040    4742 scope.go:122] "RemoveContainer" containerID="dd4187340b4f7a0450b4f62dab182f717a1b2dbef2a0450ae0468c3c9a4703da"
Oct 29 23:03:23.600355 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:23.600335    4742 scope.go:122] "RemoveContainer" containerID="1ec94ddec8eeebe2707ac9d2d10f9b8cb6254a94c203d32a8b37eea4783d26df"
Oct 29 23:03:23.607801 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:23.607783    4742 scope.go:122] "RemoveContainer" containerID="d68a11f7a71763e5cd08207cd6c4012303d140a813075e904c1c53f8feaa8f5a"
Oct 29 23:03:23.615064 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:23.615028    4742 scope.go:122] "RemoveContainer" containerID="96370aaaf20b8052a4abbbea07f8fd79595aac7fe7f31615f47c02c0013df938"
Oct 29 23:03:23.623968 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:23.623938    4742 scope.go:122] "RemoveContainer" containerID="01887eaa789371afef5a04055545f07818e9d15525f7cb796dd3a13e13bc8988"
Oct 29 23:03:28.075650 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.075618    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-848f956548-9fnqq"]
Oct 29 23:03:28.076178 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.075885    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-848f956548-9fnqq" podUID="c0aa34ce-05b1-4a1d-8280-4213c725b1e8" containerName="manager" containerID="cri-o://958757d7eb7f7c4fa255058dfb0c4657b61fbe8673d7faf28332964523a06121" gracePeriod=30
Oct 29 23:03:28.149315 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.149270    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-848f956548-9fnqq" podUID="c0aa34ce-05b1-4a1d-8280-4213c725b1e8" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.126:8081/readyz\": dial tcp 10.134.0.126:8081: connect: connection refused"
Oct 29 23:03:28.323944 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.323916    4742 generic.go:356] "Generic (PLEG): container finished" podID="c0aa34ce-05b1-4a1d-8280-4213c725b1e8" containerID="958757d7eb7f7c4fa255058dfb0c4657b61fbe8673d7faf28332964523a06121" exitCode=0
Oct 29 23:03:28.324094 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.323984    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848f956548-9fnqq" event={"ID":"c0aa34ce-05b1-4a1d-8280-4213c725b1e8","Type":"ContainerDied","Data":"958757d7eb7f7c4fa255058dfb0c4657b61fbe8673d7faf28332964523a06121"}
Oct 29 23:03:28.324094 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.324065    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848f956548-9fnqq" event={"ID":"c0aa34ce-05b1-4a1d-8280-4213c725b1e8","Type":"ContainerDied","Data":"04591a216a36f03582ccdfbfafd3a46d7bd87cc4345328ff91a34196dc83dd1f"}
Oct 29 23:03:28.324198 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.324101    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04591a216a36f03582ccdfbfafd3a46d7bd87cc4345328ff91a34196dc83dd1f"
Oct 29 23:03:28.324749 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.324727    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:03:28.438637 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.438608    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-webhook-cert\") pod \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") "
Oct 29 23:03:28.438837 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.438657    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-metrics-certs\") pod \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") "
Oct 29 23:03:28.438837 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.438745    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-kube-api-access-nhjt9\" (UniqueName: \"kubernetes.io/projected/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-kube-api-access-nhjt9\") pod \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\" (UID: \"c0aa34ce-05b1-4a1d-8280-4213c725b1e8\") "
Oct 29 23:03:28.440887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.440854    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-kube-api-access-nhjt9" pod "c0aa34ce-05b1-4a1d-8280-4213c725b1e8" (UID: "c0aa34ce-05b1-4a1d-8280-4213c725b1e8"). InnerVolumeSpecName "kube-api-access-nhjt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:28.440887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.440869    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-metrics-certs" pod "c0aa34ce-05b1-4a1d-8280-4213c725b1e8" (UID: "c0aa34ce-05b1-4a1d-8280-4213c725b1e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:28.441025 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.440850    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-webhook-cert" pod "c0aa34ce-05b1-4a1d-8280-4213c725b1e8" (UID: "c0aa34ce-05b1-4a1d-8280-4213c725b1e8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:28.540210 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.540174    4742 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-webhook-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:03:28.540210 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.540205    4742 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-metrics-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:03:28.540210 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:28.540216    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhjt9\" (UniqueName: \"kubernetes.io/projected/c0aa34ce-05b1-4a1d-8280-4213c725b1e8-kube-api-access-nhjt9\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:03:29.329321 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:29.329235    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848f956548-9fnqq"
Oct 29 23:03:29.376753 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:29.376722    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-848f956548-9fnqq"]
Oct 29 23:03:29.386912 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:29.386889    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-848f956548-9fnqq"]
Oct 29 23:03:30.568514 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568483    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-848f956548-dgrbz"]
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568808    4742 cpu_manager.go:414] "containerMap: removing container" podUID="c0aa34ce-05b1-4a1d-8280-4213c725b1e8" containerName="manager"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568826    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0aa34ce-05b1-4a1d-8280-4213c725b1e8" containerName="manager"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568845    4742 cpu_manager.go:414] "containerMap: removing container" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568851    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568859    4742 cpu_manager.go:414] "containerMap: removing container" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568865    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568881    4742 cpu_manager.go:414] "containerMap: removing container" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568885    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568894    4742 cpu_manager.go:414] "containerMap: removing container" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy"
Oct 29 23:03:30.568900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568899    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy"
Oct 29 23:03:30.569246 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568949    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0aa34ce-05b1-4a1d-8280-4213c725b1e8" containerName="manager"
Oct 29 23:03:30.569246 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568958    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9b6036-f938-43b2-8519-184c45185696" containerName="istio-proxy"
Oct 29 23:03:30.569246 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568965    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy"
Oct 29 23:03:30.569246 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568973    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4a636e-a3b0-4347-8b47-68c6b6dca509" containerName="istio-proxy"
Oct 29 23:03:30.569246 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.568979    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aea0d04-5bfa-41eb-a0ad-5b5187e5cf3a" containerName="istio-proxy"
Oct 29 23:03:30.572306 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.572289    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.596182 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.596152    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jmzz4\""
Oct 29 23:03:30.596182 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.596149    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 29 23:03:30.596182 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.596159    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 29 23:03:30.597314 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.597295    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 29 23:03:30.600852 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.600817    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848f956548-dgrbz"]
Oct 29 23:03:30.657975 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.657938    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntfn\" (UniqueName: \"kubernetes.io/projected/f966c44c-b46f-4c0a-8461-6dccde541d26-kube-api-access-qntfn\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.658182 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.658126    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f966c44c-b46f-4c0a-8461-6dccde541d26-metrics-certs\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.658182 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.658159    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f966c44c-b46f-4c0a-8461-6dccde541d26-webhook-cert\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.759001 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.758958    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntfn\" (UniqueName: \"kubernetes.io/projected/f966c44c-b46f-4c0a-8461-6dccde541d26-kube-api-access-qntfn\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.759208 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.759065    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f966c44c-b46f-4c0a-8461-6dccde541d26-metrics-certs\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.759208 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.759112    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f966c44c-b46f-4c0a-8461-6dccde541d26-webhook-cert\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.761582 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.761554    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f966c44c-b46f-4c0a-8461-6dccde541d26-webhook-cert\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.761704 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.761591    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f966c44c-b46f-4c0a-8461-6dccde541d26-metrics-certs\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.776960 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.776935    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntfn\" (UniqueName: \"kubernetes.io/projected/f966c44c-b46f-4c0a-8461-6dccde541d26-kube-api-access-qntfn\") pod \"maas-controller-848f956548-dgrbz\" (UID: \"f966c44c-b46f-4c0a-8461-6dccde541d26\") " pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.885303 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.885271    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:30.989206 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:30.989171    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0aa34ce-05b1-4a1d-8280-4213c725b1e8" path="/var/lib/kubelet/pods/c0aa34ce-05b1-4a1d-8280-4213c725b1e8/volumes"
Oct 29 23:03:31.181546 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:31.181521    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848f956548-dgrbz"]
Oct 29 23:03:31.182637 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:03:31.182613    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf966c44c_b46f_4c0a_8461_6dccde541d26.slice/crio-b387244c6ee3c0d71c3e9df739c73ad477a68e16a30a887f55fe459f3c1fdd0a WatchSource:0}: Error finding container b387244c6ee3c0d71c3e9df739c73ad477a68e16a30a887f55fe459f3c1fdd0a: Status 404 returned error can't find the container with id b387244c6ee3c0d71c3e9df739c73ad477a68e16a30a887f55fe459f3c1fdd0a
Oct 29 23:03:31.341345 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:31.341312    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848f956548-dgrbz" event={"ID":"f966c44c-b46f-4c0a-8461-6dccde541d26","Type":"ContainerStarted","Data":"b387244c6ee3c0d71c3e9df739c73ad477a68e16a30a887f55fe459f3c1fdd0a"}
Oct 29 23:03:32.354443 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:32.354406    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848f956548-dgrbz" event={"ID":"f966c44c-b46f-4c0a-8461-6dccde541d26","Type":"ContainerStarted","Data":"0f4bf183cd89c28a8d359e75a91fd9c33473de3f5058593d1c694e29c61772be"}
Oct 29 23:03:32.354917 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:32.354553    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:32.393657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:32.393595    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-848f956548-dgrbz" podStartSLOduration=1.676967346 podStartE2EDuration="2.393579253s" totalImagesPullingTime="716.611907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:32.392460914 +0000 UTC m=+1269.533202498" watchObservedRunningTime="2026-10-29 23:03:32.393579253 +0000 UTC m=+1269.534320837"
Oct 29 23:03:34.029547 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:34.029507    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"]
Oct 29 23:03:34.040479 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:34.040449    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d35c3c6-dtcmk"]
Oct 29 23:03:34.989445 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:34.989401    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d34d281-1cf8-473a-a5c5-47a7f8351ac3" path="/var/lib/kubelet/pods/4d34d281-1cf8-473a-a5c5-47a7f8351ac3/volumes"
Oct 29 23:03:41.030546 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:41.030509    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"]
Oct 29 23:03:41.040602 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:41.040573    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c941b5-bbmqn"]
Oct 29 23:03:42.990906 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:42.990872    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013e3ace-4b65-460d-8208-d58e4a8210ec" path="/var/lib/kubelet/pods/013e3ace-4b65-460d-8208-d58e4a8210ec/volumes"
Oct 29 23:03:43.362135 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:43.362099    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-848f956548-dgrbz"
Oct 29 23:03:50.027450 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:50.027411    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"]
Oct 29 23:03:50.037519 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:50.037488    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf3cf70-rfl7q"]
Oct 29 23:03:50.989386 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:03:50.989352    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0af68dc-1260-4c55-a33d-352ba320d7b9" path="/var/lib/kubelet/pods/d0af68dc-1260-4c55-a33d-352ba320d7b9/volumes"
Oct 29 23:03:53.122783 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:03:53.122743    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:04:17.035546 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:17.035458    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"]
Oct 29 23:04:17.045840 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:17.045804    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24754ef4-wd6b6"]
Oct 29 23:04:18.989918 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:18.989882    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22bfd7a1-949d-46ff-ba4f-f1d2d831ef68" path="/var/lib/kubelet/pods/22bfd7a1-949d-46ff-ba4f-f1d2d831ef68/volumes"
Oct 29 23:04:23.123657 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:04:23.123616    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:04:23.694398 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:23.694363    4742 scope.go:122] "RemoveContainer" containerID="66673b571b75a680404a931a13a93c45a4f5ecd8857692fb2eea13916349bf04"
Oct 29 23:04:23.705234 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:23.705208    4742 scope.go:122] "RemoveContainer" containerID="1d33611f30fdd5e9dc9007b4e66faeab643fa3684f18ffb1ead1a3540991a147"
Oct 29 23:04:23.715568 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:23.715539    4742 scope.go:122] "RemoveContainer" containerID="8f50ca82b11f16dd2552442f7b267eec556be06e7366964ecb7b260880596740"
Oct 29 23:04:23.729793 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:23.729770    4742 scope.go:122] "RemoveContainer" containerID="1612ebfe60b884aebaa56f9e2ac838a9daae0d66e29da53a1f609ea3466599c0"
Oct 29 23:04:23.740641 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:23.740615    4742 scope.go:122] "RemoveContainer" containerID="a875c0ecd6390143f7b31788a625678d519299ad855e2edc82adef6f06d0239a"
Oct 29 23:04:23.750252 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:23.750223    4742 scope.go:122] "RemoveContainer" containerID="b74b4f57f17ba645652746e461d99afebe1bc25452ffc1515bc2c8406fdc5c32"
Oct 29 23:04:23.759983 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:23.759952    4742 scope.go:122] "RemoveContainer" containerID="7b68d3a6b90db0b2e8da6ff441c909a688f532616cd8da832df0432d68405d0a"
Oct 29 23:04:23.770843 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:04:23.770816    4742 scope.go:122] "RemoveContainer" containerID="951ba2d476139e794c120631d754eec661ee96fd249576660c89b527893ef94a"
Oct 29 23:04:53.122436 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:04:53.122398    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:05:23.122825 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:05:23.122772    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:05:23.833106 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:23.833068    4742 scope.go:122] "RemoveContainer" containerID="527fc4f43d9d81399af6e54d6fc6da702d4f424f00daeb5ec274244275c456ba"
Oct 29 23:05:23.841658 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:23.841639    4742 scope.go:122] "RemoveContainer" containerID="e99d53cf0663fee88172c2227ca7bc0f929abefcc1993e6ecee36a3e4ccfee75"
Oct 29 23:05:23.849339 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:23.849316    4742 scope.go:122] "RemoveContainer" containerID="016be460ff311bc344c29bc4222e9383019bdbb95918f667afff9ff37bbf2096"
Oct 29 23:05:23.857032 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:23.857011    4742 scope.go:122] "RemoveContainer" containerID="87a8aaff55f83a73eb7ec5389206b54bd5ad4dc577fb8fc2fcf399542905f65e"
Oct 29 23:05:23.864359 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:23.864339    4742 scope.go:122] "RemoveContainer" containerID="e23640795f556d68be41f43e5dbefd5e1c01081ee422f42b7ef21cbd733416a7"
Oct 29 23:05:39.026418 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:39.026338    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"]
Oct 29 23:05:39.036991 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:39.036957    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"]
Oct 29 23:05:39.038416 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:39.038394    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67c1321f-84nz5"]
Oct 29 23:05:39.039753 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:39.039734    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9edb9646-m4v49"]
Oct 29 23:05:40.989316 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:40.989275    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e73e4ee-008e-43bd-867d-106f11547d5a" path="/var/lib/kubelet/pods/6e73e4ee-008e-43bd-867d-106f11547d5a/volumes"
Oct 29 23:05:40.989863 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:40.989842    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15f3912-65fe-4d9f-8b45-3da75b4c56d3" path="/var/lib/kubelet/pods/a15f3912-65fe-4d9f-8b45-3da75b4c56d3/volumes"
Oct 29 23:05:44.024538 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:44.024503    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"]
Oct 29 23:05:44.034706 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:44.034675    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8115d7d-b6rbd"]
Oct 29 23:05:44.990091 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:44.990044    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3227003a-fe22-4483-bd7f-20a9d0cc6a5b" path="/var/lib/kubelet/pods/3227003a-fe22-4483-bd7f-20a9d0cc6a5b/volumes"
Oct 29 23:05:45.035312 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:45.035279    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"]
Oct 29 23:05:45.045643 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:45.045618    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa5dc33-9tnhx"]
Oct 29 23:05:46.988929 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:46.988898    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42fa266-4ef0-4806-b8d1-dabcbcf8d126" path="/var/lib/kubelet/pods/e42fa266-4ef0-4806-b8d1-dabcbcf8d126/volumes"
Oct 29 23:05:49.026284 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:49.026251    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"]
Oct 29 23:05:49.036174 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:49.036150    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ac7de2-5q777"]
Oct 29 23:05:50.989912 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:05:50.989877    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0abd2b2-5442-409c-ac11-ff9491ddb54b" path="/var/lib/kubelet/pods/d0abd2b2-5442-409c-ac11-ff9491ddb54b/volumes"
Oct 29 23:05:53.131032 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:05:53.130990    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:06:00.043529 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:06:00.043495    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"]
Oct 29 23:06:00.053538 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:06:00.053502    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"]
Oct 29 23:06:00.055134 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:06:00.055109    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de552c8-2mvv7"]
Oct 29 23:06:00.056307 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:06:00.056285    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda17b4e-tfqc2"]
Oct 29 23:06:00.989760 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:06:00.989729    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42786f94-d2b2-493c-b722-2c0ef835e650" path="/var/lib/kubelet/pods/42786f94-d2b2-493c-b722-2c0ef835e650/volumes"
Oct 29 23:06:00.990145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:06:00.990131    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08b4d2b-445e-4ae7-a976-419fc19ec87a" path="/var/lib/kubelet/pods/a08b4d2b-445e-4ae7-a976-419fc19ec87a/volumes"
Oct 29 23:06:23.121891 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:06:23.121854    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:06:23.897603 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:06:23.897571    4742 scope.go:122] "RemoveContainer" containerID="a6d5a873cb2fb51dc85a38d22f10568eee911517026a76f597f3320e96f22648"
Oct 29 23:06:53.122575 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:06:53.122533    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:07:23.121924 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:07:23.121883    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:07:23.919967 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.919937    4742 scope.go:122] "RemoveContainer" containerID="ee33569cd7d2153d07876c4bda581ab3d7bf9bdc398a16d7791334cdb72d21f6"
Oct 29 23:07:23.927970 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.927951    4742 scope.go:122] "RemoveContainer" containerID="1133bf8f05d5205c1106d5182f71715cbb16e54afdc06b105895bff503faceb8"
Oct 29 23:07:23.934951 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.934933    4742 scope.go:122] "RemoveContainer" containerID="f3815852135929186087bd2fd09733ed1552d0b4042b51033e82b0a7ceb1e930"
Oct 29 23:07:23.941732 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.941715    4742 scope.go:122] "RemoveContainer" containerID="1ecdbe2acd710b6a7c47928d1a845a26385017374766426f84b7714d84d08e5e"
Oct 29 23:07:23.948723 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.948705    4742 scope.go:122] "RemoveContainer" containerID="8534d27dac7c457360cdb347fa4a67ade08a37cd93c48c5af708e62e52682cfb"
Oct 29 23:07:23.955361 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.955344    4742 scope.go:122] "RemoveContainer" containerID="f9917f26e3b1e1144a0c27a6d22e476cd9fee94b983134b5dd3687f09306f86a"
Oct 29 23:07:23.961898 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.961883    4742 scope.go:122] "RemoveContainer" containerID="59d199dd17337f2e6a84c15d5472d20a15148f23db1ea1410ebd05f39b3bee19"
Oct 29 23:07:23.968374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.968357    4742 scope.go:122] "RemoveContainer" containerID="d169e44df1bc43d327a71f247b77b613098dd12bc5f79bd8c705ae5f6db5b38f"
Oct 29 23:07:23.974773 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:23.974758    4742 scope.go:122] "RemoveContainer" containerID="b499c4b2169e6793ed18d0789958711fcffeef4f9a52de800c7cce72496486a4"
Oct 29 23:07:30.046779 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:30.046745    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"]
Oct 29 23:07:30.060266 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:30.060236    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"]
Oct 29 23:07:30.061502 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:30.061478    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cde6278-v46ct"]
Oct 29 23:07:30.071112 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:30.071077    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ecc37f-vcd4m"]
Oct 29 23:07:30.989111 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:30.989063    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ed274b-9c66-4785-8a63-e3e47304148d" path="/var/lib/kubelet/pods/64ed274b-9c66-4785-8a63-e3e47304148d/volumes"
Oct 29 23:07:30.989474 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:30.989459    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a505dd-0531-43fe-8b46-1084006009c1" path="/var/lib/kubelet/pods/f4a505dd-0531-43fe-8b46-1084006009c1/volumes"
Oct 29 23:07:53.121532 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:07:53.121488    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:07:58.010806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.010771    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"]
Oct 29 23:07:58.014638 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.014610    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.027812 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.027789    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2b2738-openshift-default-dockercfg-xhs7r\""
Oct 29 23:07:58.041422 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.041392    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"]
Oct 29 23:07:58.116943 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.116907    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-certs\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.117147 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.116958    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-envoy\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.117147 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.117023    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-socket\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.117147 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.117055    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-data\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.117147 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.117075    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad010e63-a217-403f-b132-908263363c9f-istio-podinfo\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.117319 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.117166    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-istio-token\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.117319 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.117212    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad010e63-a217-403f-b132-908263363c9f-istiod-ca-cert\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.117319 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.117236    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-credential-socket\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.117319 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.117288    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9v2\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-kube-api-access-nc9v2\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.218687 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.218653    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-data\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.218687 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.218696    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad010e63-a217-403f-b132-908263363c9f-istio-podinfo\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.218952 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.218824    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-istio-token\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.218952 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.218858    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad010e63-a217-403f-b132-908263363c9f-istiod-ca-cert\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.218952 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.218883    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-credential-socket\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219156 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.218954    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9v2\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-kube-api-access-nc9v2\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219156 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.219025    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-certs\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219156 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.219070    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-data\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219334 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.219226    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-envoy\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219334 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.219264    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-socket\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219334 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.219280    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-credential-socket\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219495 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.219419    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-certs\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219536 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.219514    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-socket\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.219631 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.219603    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad010e63-a217-403f-b132-908263363c9f-istiod-ca-cert\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.221263 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.221242    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad010e63-a217-403f-b132-908263363c9f-istio-podinfo\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.221372 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.221291    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-envoy\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.237919 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.237889    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-istio-token\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.238374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.238352    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9v2\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-kube-api-access-nc9v2\") pod \"e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") " pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.327449 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.327359    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:58.635344 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.635321    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"]
Oct 29 23:07:58.637302 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:07:58.637276    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad010e63_a217_403f_b132_908263363c9f.slice/crio-afcf6f68e4f5a9707996c2a19489715c9e9fa422873a5212821b0cf5897e0a23 WatchSource:0}: Error finding container afcf6f68e4f5a9707996c2a19489715c9e9fa422873a5212821b0cf5897e0a23: Status 404 returned error can't find the container with id afcf6f68e4f5a9707996c2a19489715c9e9fa422873a5212821b0cf5897e0a23
Oct 29 23:07:58.639378 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.639349    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:07:58.639464 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.639410    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:07:58.639464 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.639438    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:07:58.739173 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.739134    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" event={"ID":"ad010e63-a217-403f-b132-908263363c9f","Type":"ContainerStarted","Data":"30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e"}
Oct 29 23:07:58.739369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.739198    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" event={"ID":"ad010e63-a217-403f-b132-908263363c9f","Type":"ContainerStarted","Data":"afcf6f68e4f5a9707996c2a19489715c9e9fa422873a5212821b0cf5897e0a23"}
Oct 29 23:07:58.779871 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:58.779811    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" podStartSLOduration=1.7797915770000001 podStartE2EDuration="1.779791577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:58.779101574 +0000 UTC m=+1535.919843153" watchObservedRunningTime="2026-10-29 23:07:58.779791577 +0000 UTC m=+1535.920533166"
Oct 29 23:07:59.328453 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:59.328408    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:59.333360 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:59.333335    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:59.743465 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:59.743434    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:07:59.744387 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:07:59.744366    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:08:03.078797 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.078765    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"]
Oct 29 23:08:03.082260 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.082242    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.095596 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.095567    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2b2738\""
Oct 29 23:08:03.107763 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.107730    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"]
Oct 29 23:08:03.140506 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.140463    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-daa70505-dockercfg-tvctm\""
Oct 29 23:08:03.162580 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.162545    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwk2\" (UniqueName: \"kubernetes.io/projected/561f1fd1-81db-4d31-a3e9-568814146c5a-kube-api-access-dbwk2\") pod \"payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb\" (UID: \"561f1fd1-81db-4d31-a3e9-568814146c5a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.162777 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.162609    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561f1fd1-81db-4d31-a3e9-568814146c5a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb\" (UID: \"561f1fd1-81db-4d31-a3e9-568814146c5a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.263280 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.263238    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561f1fd1-81db-4d31-a3e9-568814146c5a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb\" (UID: \"561f1fd1-81db-4d31-a3e9-568814146c5a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.263460 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.263331    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwk2\" (UniqueName: \"kubernetes.io/projected/561f1fd1-81db-4d31-a3e9-568814146c5a-kube-api-access-dbwk2\") pod \"payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb\" (UID: \"561f1fd1-81db-4d31-a3e9-568814146c5a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.263925 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.263903    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561f1fd1-81db-4d31-a3e9-568814146c5a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb\" (UID: \"561f1fd1-81db-4d31-a3e9-568814146c5a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.282148 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.282111    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwk2\" (UniqueName: \"kubernetes.io/projected/561f1fd1-81db-4d31-a3e9-568814146c5a-kube-api-access-dbwk2\") pod \"payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb\" (UID: \"561f1fd1-81db-4d31-a3e9-568814146c5a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.395008 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.394971    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.688714 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.688640    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"]
Oct 29 23:08:03.692889 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:08:03.692859    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561f1fd1_81db_4d31_a3e9_568814146c5a.slice/crio-5a7498c28f2d148838d80f0545d55afa357ed19204b118a539400384e88e779f WatchSource:0}: Error finding container 5a7498c28f2d148838d80f0545d55afa357ed19204b118a539400384e88e779f: Status 404 returned error can't find the container with id 5a7498c28f2d148838d80f0545d55afa357ed19204b118a539400384e88e779f
Oct 29 23:08:03.761123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.761063    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb" event={"ID":"561f1fd1-81db-4d31-a3e9-568814146c5a","Type":"ContainerStarted","Data":"c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687"}
Oct 29 23:08:03.761295 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.761142    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb" event={"ID":"561f1fd1-81db-4d31-a3e9-568814146c5a","Type":"ContainerStarted","Data":"5a7498c28f2d148838d80f0545d55afa357ed19204b118a539400384e88e779f"}
Oct 29 23:08:03.761295 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.761227    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:03.795818 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:03.795762    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb" podStartSLOduration=0.795748198 podStartE2EDuration="795.748198ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:03.795403075 +0000 UTC m=+1540.936144660" watchObservedRunningTime="2026-10-29 23:08:03.795748198 +0000 UTC m=+1540.936489775"
Oct 29 23:08:14.768213 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:14.768179    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:08:15.786248 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:15.786192    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:08:15.786667 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:15.786292    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:08:16.848391 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:16.848358    4742 generic.go:356] "Generic (PLEG): container finished" podID="ad010e63-a217-403f-b132-908263363c9f" containerID="30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e" exitCode=0
Oct 29 23:08:16.848825 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:16.848433    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" event={"ID":"ad010e63-a217-403f-b132-908263363c9f","Type":"ContainerDied","Data":"30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e"}
Oct 29 23:08:16.848825 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:16.848803    4742 scope.go:122] "RemoveContainer" containerID="30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e"
Oct 29 23:08:16.849906 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:16.849877    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:08:16.850014 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:16.849959    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:08:16.850014 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:16.850007    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:08:17.855179 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:17.855146    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" event={"ID":"ad010e63-a217-403f-b132-908263363c9f","Type":"ContainerStarted","Data":"4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338"}
Oct 29 23:08:18.328622 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.328543    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:08:18.328771 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.328716    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:08:18.328815 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.328767    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 29 23:08:18.521556 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.521522    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"]
Oct 29 23:08:18.524049 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.524028    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.537473 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.537449    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2b2738\"/\"default-dockercfg-f8zq6\""
Oct 29 23:08:18.537473 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.537459    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2b2738\"/\"kube-root-ca.crt\""
Oct 29 23:08:18.547280 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.547250    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2b2738\"/\"openshift-service-ca.crt\""
Oct 29 23:08:18.547280 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.547268    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2b2738\"/\"simulator-single-main-2b2738-kserve-self-signed-certs\""
Oct 29 23:08:18.549844 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.549822    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"]
Oct 29 23:08:18.638202 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.638171    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-model-cache\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.638374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.638286    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tmp-dir\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.638374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.638335    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tls-certs\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.638374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.638361    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-home\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.638480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.638447    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-dshm\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.638517 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.638500    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gwmh\" (UniqueName: \"kubernetes.io/projected/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-kube-api-access-9gwmh\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.739532 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.739499    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-home\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.739696 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.739543    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-dshm\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.739696 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.739572    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gwmh\" (UniqueName: \"kubernetes.io/projected/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-kube-api-access-9gwmh\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.739696 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.739621    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-model-cache\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.739696 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.739682    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tmp-dir\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.739905 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.739704    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tls-certs\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.740000 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.739972    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-home\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.740065 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.740002    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-model-cache\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.740166 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.740126    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tmp-dir\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.741854 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.741828    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-dshm\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.742044 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.742029    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tls-certs\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.776127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.765849    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gwmh\" (UniqueName: \"kubernetes.io/projected/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-kube-api-access-9gwmh\") pod \"simulator-single-main-2b2738-kserve-7546d58cfd-xglnz\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") " pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:18.835533 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:18.835502    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:19.182151 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:19.179152    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"]
Oct 29 23:08:19.328429 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:19.328394    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:08:19.328590 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:19.328449    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 29 23:08:19.892468 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:19.892434    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz" event={"ID":"3d27c127-787d-4c02-8b5b-8f33a2dd9eee","Type":"ContainerStarted","Data":"7940e2b0154d89b1fdfb46a69643e9e33e651d735c2eddc9a7c6af2636b9e735"}
Oct 29 23:08:19.892696 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:19.892478    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz" event={"ID":"3d27c127-787d-4c02-8b5b-8f33a2dd9eee","Type":"ContainerStarted","Data":"272a1a5e7582b57d6879bce1924fa8f848fb38d8f0c0419346a5a346bcfe0d9f"}
Oct 29 23:08:19.929424 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:19.929373    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz" podStartSLOduration=1.929358856 podStartE2EDuration="1.929358856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:19.927796391 +0000 UTC m=+1557.068537980" watchObservedRunningTime="2026-10-29 23:08:19.929358856 +0000 UTC m=+1557.070100439"
Oct 29 23:08:20.328506 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:20.328422    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:08:20.328506 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:20.328477    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 29 23:08:20.836360 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:20.836326    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:21.329605 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:21.329519    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:08:21.330170 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:21.330134    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 29 23:08:22.327672 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:22.327635    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:08:22.327856 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:22.327689    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 29 23:08:23.122845 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:08:23.122802    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:08:23.404262 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:23.404173    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:08:23.404753 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:23.404730    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:08:23.405622 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:23.405599    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:08:24.018215 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:24.018189    4742 scope.go:122] "RemoveContainer" containerID="a770fe1cf82f9e5d883afeb1d2421a57d576f9d1843619bca3a086141b0d3e51"
Oct 29 23:08:28.836136 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:28.836069    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:28.848293 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:28.848268    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:28.944386 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:28.944361    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:08:41.848347 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.848268    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"]
Oct 29 23:08:41.853232 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.853211    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:41.869960 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.869938    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"]
Oct 29 23:08:41.879688 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.879663    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:08:41.879850 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.879719    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:08:41.879850 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.879772    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:08:41.879850 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.879786    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:08:41.973145 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.973111    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:41.973309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.973152    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4gb\" (UniqueName: \"kubernetes.io/projected/939ddfed-2017-445c-bd1e-6d8e222a348a-kube-api-access-xw4gb\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:41.973309 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:41.973258    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:42.074051 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.074017    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:42.074051 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.074054    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4gb\" (UniqueName: \"kubernetes.io/projected/939ddfed-2017-445c-bd1e-6d8e222a348a-kube-api-access-xw4gb\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:42.074299 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.074202    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:42.074798 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.074774    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:42.076293 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.076271    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:42.093351 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.093323    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4gb\" (UniqueName: \"kubernetes.io/projected/939ddfed-2017-445c-bd1e-6d8e222a348a-kube-api-access-xw4gb\") pod \"maas-api-revoke-sub-9860ca44-ldxwn\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:42.165964 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.165891    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:42.456100 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.456061    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"]
Oct 29 23:08:42.458563 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:08:42.458535    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939ddfed_2017_445c_bd1e_6d8e222a348a.slice/crio-2e1f5f883f376be1af2f0d923d71220c666878a4e1cf1c474a8f758a038aae67 WatchSource:0}: Error finding container 2e1f5f883f376be1af2f0d923d71220c666878a4e1cf1c474a8f758a038aae67: Status 404 returned error can't find the container with id 2e1f5f883f376be1af2f0d923d71220c666878a4e1cf1c474a8f758a038aae67
Oct 29 23:08:42.998449 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.998418    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn" event={"ID":"939ddfed-2017-445c-bd1e-6d8e222a348a","Type":"ContainerStarted","Data":"a2a0080defba6a9bafd5c91282d22e4f63f037e17c141c00058fc7bcf44d9a93"}
Oct 29 23:08:42.998828 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:42.998459    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn" event={"ID":"939ddfed-2017-445c-bd1e-6d8e222a348a","Type":"ContainerStarted","Data":"2e1f5f883f376be1af2f0d923d71220c666878a4e1cf1c474a8f758a038aae67"}
Oct 29 23:08:43.034029 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:43.033945    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn" podStartSLOduration=2.033931054 podStartE2EDuration="2.033931054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:43.032737409 +0000 UTC m=+1580.173478994" watchObservedRunningTime="2026-10-29 23:08:43.033931054 +0000 UTC m=+1580.174672639"
Oct 29 23:08:44.004983 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:44.004948    4742 generic.go:356] "Generic (PLEG): container finished" podID="939ddfed-2017-445c-bd1e-6d8e222a348a" containerID="a2a0080defba6a9bafd5c91282d22e4f63f037e17c141c00058fc7bcf44d9a93" exitCode=0
Oct 29 23:08:44.005447 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:44.005039    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn" event={"ID":"939ddfed-2017-445c-bd1e-6d8e222a348a","Type":"ContainerDied","Data":"a2a0080defba6a9bafd5c91282d22e4f63f037e17c141c00058fc7bcf44d9a93"}
Oct 29 23:08:45.149446 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.149423    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:45.204657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.204631    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-openshift-ca\") pod \"939ddfed-2017-445c-bd1e-6d8e222a348a\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") "
Oct 29 23:08:45.204821 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.204670    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-xks-ca\") pod \"939ddfed-2017-445c-bd1e-6d8e222a348a\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") "
Oct 29 23:08:45.204821 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.204714    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939ddfed-2017-445c-bd1e-6d8e222a348a-kube-api-access-xw4gb\" (UniqueName: \"kubernetes.io/projected/939ddfed-2017-445c-bd1e-6d8e222a348a-kube-api-access-xw4gb\") pod \"939ddfed-2017-445c-bd1e-6d8e222a348a\" (UID: \"939ddfed-2017-445c-bd1e-6d8e222a348a\") "
Oct 29 23:08:45.205134 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.205064    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-openshift-ca" pod "939ddfed-2017-445c-bd1e-6d8e222a348a" (UID: "939ddfed-2017-445c-bd1e-6d8e222a348a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:45.206853 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.206825    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-xks-ca" pod "939ddfed-2017-445c-bd1e-6d8e222a348a" (UID: "939ddfed-2017-445c-bd1e-6d8e222a348a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:45.206981 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.206952    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939ddfed-2017-445c-bd1e-6d8e222a348a-kube-api-access-xw4gb" pod "939ddfed-2017-445c-bd1e-6d8e222a348a" (UID: "939ddfed-2017-445c-bd1e-6d8e222a348a"). InnerVolumeSpecName "kube-api-access-xw4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:45.306159 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.306071    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.306159 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.306117    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939ddfed-2017-445c-bd1e-6d8e222a348a-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.306159 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:45.306128    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw4gb\" (UniqueName: \"kubernetes.io/projected/939ddfed-2017-445c-bd1e-6d8e222a348a-kube-api-access-xw4gb\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:08:46.015338 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:46.015308    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"
Oct 29 23:08:46.015539 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:46.015345    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn" event={"ID":"939ddfed-2017-445c-bd1e-6d8e222a348a","Type":"ContainerDied","Data":"2e1f5f883f376be1af2f0d923d71220c666878a4e1cf1c474a8f758a038aae67"}
Oct 29 23:08:46.015539 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:46.015391    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1f5f883f376be1af2f0d923d71220c666878a4e1cf1c474a8f758a038aae67"
Oct 29 23:08:53.123047 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:08:53.123012    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:08:53.145781 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.145204    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"]
Oct 29 23:08:53.145781 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.145704    4742 cpu_manager.go:414] "containerMap: removing container" podUID="939ddfed-2017-445c-bd1e-6d8e222a348a" containerName="revoke-keys"
Oct 29 23:08:53.145781 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.145719    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939ddfed-2017-445c-bd1e-6d8e222a348a" containerName="revoke-keys"
Oct 29 23:08:53.146116 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.145829    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="939ddfed-2017-445c-bd1e-6d8e222a348a" containerName="revoke-keys"
Oct 29 23:08:53.149414 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.149390    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.162532 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.162512    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:08:53.162634 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.162579    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:08:53.162680 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.162630    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:08:53.172899 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.172876    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:08:53.174749 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.174718    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"]
Oct 29 23:08:53.280203 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.280173    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.280388 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.280226    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.280388 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.280335    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbpm\" (UniqueName: \"kubernetes.io/projected/e0aa1662-4655-4599-bb90-2166d581af0b-kube-api-access-mmbpm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.381119 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.381010    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.381119 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.381107    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbpm\" (UniqueName: \"kubernetes.io/projected/e0aa1662-4655-4599-bb90-2166d581af0b-kube-api-access-mmbpm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.381347 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.381228    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.381829 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.381805    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.383320 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.383300    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.405349 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.405323    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbpm\" (UniqueName: \"kubernetes.io/projected/e0aa1662-4655-4599-bb90-2166d581af0b-kube-api-access-mmbpm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.462285 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.462259    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:53.760938 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:53.760888    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"]
Oct 29 23:08:53.763852 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:08:53.763820    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aa1662_4655_4599_bb90_2166d581af0b.slice/crio-39c7f107a23ac0ab5ca0e785ee207c2bea510234388087ad6cbe2d3b240b3e4c WatchSource:0}: Error finding container 39c7f107a23ac0ab5ca0e785ee207c2bea510234388087ad6cbe2d3b240b3e4c: Status 404 returned error can't find the container with id 39c7f107a23ac0ab5ca0e785ee207c2bea510234388087ad6cbe2d3b240b3e4c
Oct 29 23:08:54.050535 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:54.050466    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz" event={"ID":"e0aa1662-4655-4599-bb90-2166d581af0b","Type":"ContainerStarted","Data":"39c7f107a23ac0ab5ca0e785ee207c2bea510234388087ad6cbe2d3b240b3e4c"}
Oct 29 23:08:55.056501 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:55.056414    4742 generic.go:356] "Generic (PLEG): container finished" podID="e0aa1662-4655-4599-bb90-2166d581af0b" containerID="877e240770147b1e30d89587d62241b964ae2b0f8e1dd06aa2afeeec997db7bd" exitCode=0
Oct 29 23:08:55.056501 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:55.056471    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz" event={"ID":"e0aa1662-4655-4599-bb90-2166d581af0b","Type":"ContainerDied","Data":"877e240770147b1e30d89587d62241b964ae2b0f8e1dd06aa2afeeec997db7bd"}
Oct 29 23:08:56.191104 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.191063    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:56.308378 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.308346    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-openshift-ca\") pod \"e0aa1662-4655-4599-bb90-2166d581af0b\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") "
Oct 29 23:08:56.308549 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.308398    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0aa1662-4655-4599-bb90-2166d581af0b-kube-api-access-mmbpm\" (UniqueName: \"kubernetes.io/projected/e0aa1662-4655-4599-bb90-2166d581af0b-kube-api-access-mmbpm\") pod \"e0aa1662-4655-4599-bb90-2166d581af0b\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") "
Oct 29 23:08:56.308549 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.308425    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-xks-ca\") pod \"e0aa1662-4655-4599-bb90-2166d581af0b\" (UID: \"e0aa1662-4655-4599-bb90-2166d581af0b\") "
Oct 29 23:08:56.308719 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.308698    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-openshift-ca" pod "e0aa1662-4655-4599-bb90-2166d581af0b" (UID: "e0aa1662-4655-4599-bb90-2166d581af0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:56.308801 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.308788    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.310486 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.310457    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0aa1662-4655-4599-bb90-2166d581af0b-kube-api-access-mmbpm" pod "e0aa1662-4655-4599-bb90-2166d581af0b" (UID: "e0aa1662-4655-4599-bb90-2166d581af0b"). InnerVolumeSpecName "kube-api-access-mmbpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:56.310597 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.310518    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-xks-ca" pod "e0aa1662-4655-4599-bb90-2166d581af0b" (UID: "e0aa1662-4655-4599-bb90-2166d581af0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:56.410151 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.410114    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbpm\" (UniqueName: \"kubernetes.io/projected/e0aa1662-4655-4599-bb90-2166d581af0b-kube-api-access-mmbpm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.410151 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:56.410145    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0aa1662-4655-4599-bb90-2166d581af0b-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:08:57.066602 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:57.066578    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"
Oct 29 23:08:57.066762 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:57.066612    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz" event={"ID":"e0aa1662-4655-4599-bb90-2166d581af0b","Type":"ContainerDied","Data":"39c7f107a23ac0ab5ca0e785ee207c2bea510234388087ad6cbe2d3b240b3e4c"}
Oct 29 23:08:57.066762 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:57.066660    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c7f107a23ac0ab5ca0e785ee207c2bea510234388087ad6cbe2d3b240b3e4c"
Oct 29 23:08:57.936323 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:08:57.936285    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"]
Oct 29 23:09:03.596603 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:03.596570    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"]
Oct 29 23:09:07.224246 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:07.224206    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"]
Oct 29 23:09:07.235363 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:07.235328    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2b2738-8gzhz"]
Oct 29 23:09:08.597896 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:08.597817    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb" podUID="561f1fd1-81db-4d31-a3e9-568814146c5a" containerName="payload-processing" containerID="cri-o://c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687" gracePeriod=25
Oct 29 23:09:08.990290 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:08.990197    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0aa1662-4655-4599-bb90-2166d581af0b" path="/var/lib/kubelet/pods/e0aa1662-4655-4599-bb90-2166d581af0b/volumes"
Oct 29 23:09:12.956225 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:12.956178    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz" podUID="3d27c127-787d-4c02-8b5b-8f33a2dd9eee" containerName="main" containerID="cri-o://7940e2b0154d89b1fdfb46a69643e9e33e651d735c2eddc9a7c6af2636b9e735" gracePeriod=45
Oct 29 23:09:13.145022 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.144973    4742 generic.go:356] "Generic (PLEG): container finished" podID="3d27c127-787d-4c02-8b5b-8f33a2dd9eee" containerID="7940e2b0154d89b1fdfb46a69643e9e33e651d735c2eddc9a7c6af2636b9e735" exitCode=0
Oct 29 23:09:13.145215 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.145063    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz" event={"ID":"3d27c127-787d-4c02-8b5b-8f33a2dd9eee","Type":"ContainerDied","Data":"7940e2b0154d89b1fdfb46a69643e9e33e651d735c2eddc9a7c6af2636b9e735"}
Oct 29 23:09:13.198646 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.198621    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:09:13.271230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271154    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tmp-dir\") pod \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") "
Oct 29 23:09:13.271390 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271239    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-kube-api-access-9gwmh\" (UniqueName: \"kubernetes.io/projected/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-kube-api-access-9gwmh\") pod \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") "
Oct 29 23:09:13.271390 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271263    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-model-cache\") pod \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") "
Oct 29 23:09:13.271390 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271310    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-dshm\") pod \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") "
Oct 29 23:09:13.271390 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271331    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tls-certs\") pod \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") "
Oct 29 23:09:13.271390 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271358    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-home\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-home\") pod \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\" (UID: \"3d27c127-787d-4c02-8b5b-8f33a2dd9eee\") "
Oct 29 23:09:13.271661 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271449    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tmp-dir" pod "3d27c127-787d-4c02-8b5b-8f33a2dd9eee" (UID: "3d27c127-787d-4c02-8b5b-8f33a2dd9eee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:13.271661 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271573    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-model-cache" pod "3d27c127-787d-4c02-8b5b-8f33a2dd9eee" (UID: "3d27c127-787d-4c02-8b5b-8f33a2dd9eee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:13.271661 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271633    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.271661 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.271637    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-home" pod "3d27c127-787d-4c02-8b5b-8f33a2dd9eee" (UID: "3d27c127-787d-4c02-8b5b-8f33a2dd9eee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:13.273761 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.273730    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-kube-api-access-9gwmh" pod "3d27c127-787d-4c02-8b5b-8f33a2dd9eee" (UID: "3d27c127-787d-4c02-8b5b-8f33a2dd9eee"). InnerVolumeSpecName "kube-api-access-9gwmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:13.273876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.273781    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tls-certs" pod "3d27c127-787d-4c02-8b5b-8f33a2dd9eee" (UID: "3d27c127-787d-4c02-8b5b-8f33a2dd9eee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:13.273876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.273806    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-dshm" pod "3d27c127-787d-4c02-8b5b-8f33a2dd9eee" (UID: "3d27c127-787d-4c02-8b5b-8f33a2dd9eee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:13.372469 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.372436    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.372469 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.372467    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.372679 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.372481    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.372679 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.372494    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gwmh\" (UniqueName: \"kubernetes.io/projected/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-kube-api-access-9gwmh\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.372679 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:13.372507    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d27c127-787d-4c02-8b5b-8f33a2dd9eee-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:14.159123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.151105    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"
Oct 29 23:09:14.159123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.151127    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz" event={"ID":"3d27c127-787d-4c02-8b5b-8f33a2dd9eee","Type":"ContainerDied","Data":"272a1a5e7582b57d6879bce1924fa8f848fb38d8f0c0419346a5a346bcfe0d9f"}
Oct 29 23:09:14.159123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.151179    4742 scope.go:122] "RemoveContainer" containerID="7940e2b0154d89b1fdfb46a69643e9e33e651d735c2eddc9a7c6af2636b9e735"
Oct 29 23:09:14.159123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.153369    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"]
Oct 29 23:09:14.159123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.153586    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy" containerID="cri-o://4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338" gracePeriod=30
Oct 29 23:09:14.199993 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.199964    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"]
Oct 29 23:09:14.203977 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.203956    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2b2738/simulator-single-main-2b2738-kserve-7546d58cfd-xglnz"]
Oct 29 23:09:14.766908 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.766869    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb" podUID="561f1fd1-81db-4d31-a3e9-568814146c5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:09:14.989499 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:14.989467    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d27c127-787d-4c02-8b5b-8f33a2dd9eee" path="/var/lib/kubelet/pods/3d27c127-787d-4c02-8b5b-8f33a2dd9eee/volumes"
Oct 29 23:09:19.395402 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.395374    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:09:19.430125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430014    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-data\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430052    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-certs\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430079    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-istio-token\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-istio-token\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430146    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-credential-socket\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430267    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad010e63-a217-403f-b132-908263363c9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad010e63-a217-403f-b132-908263363c9f-istio-podinfo\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430371    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-data" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:19.430407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430371    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-socket\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430407 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430384    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-certs" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:19.430680 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430447    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad010e63-a217-403f-b132-908263363c9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad010e63-a217-403f-b132-908263363c9f-istiod-ca-cert\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430680 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430510    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-envoy\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430680 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430531    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-credential-socket" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:19.430680 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430560    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-kube-api-access-nc9v2\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-kube-api-access-nc9v2\") pod \"ad010e63-a217-403f-b132-908263363c9f\" (UID: \"ad010e63-a217-403f-b132-908263363c9f\") "
Oct 29 23:09:19.430680 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430570    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-socket" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:19.430990 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430860    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad010e63-a217-403f-b132-908263363c9f-istiod-ca-cert" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:19.430990 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430969    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.430990 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.430989    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.431177 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.431003    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.431177 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.431016    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.431177 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.431029    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad010e63-a217-403f-b132-908263363c9f-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.432529 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.432499    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad010e63-a217-403f-b132-908263363c9f-istio-podinfo" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:09:19.432657 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.432619    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-kube-api-access-nc9v2" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "kube-api-access-nc9v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:19.432881 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.432860    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-envoy" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:19.432881 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.432874    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-istio-token" pod "ad010e63-a217-403f-b132-908263363c9f" (UID: "ad010e63-a217-403f-b132-908263363c9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:19.531965 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.531934    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.531965 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.531962    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad010e63-a217-403f-b132-908263363c9f-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.531965 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.531971    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad010e63-a217-403f-b132-908263363c9f-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.532233 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:19.531981    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc9v2\" (UniqueName: \"kubernetes.io/projected/ad010e63-a217-403f-b132-908263363c9f-kube-api-access-nc9v2\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:20.181535 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.181503    4742 generic.go:356] "Generic (PLEG): container finished" podID="ad010e63-a217-403f-b132-908263363c9f" containerID="4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338" exitCode=0
Oct 29 23:09:20.181721 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.181556    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"
Oct 29 23:09:20.181721 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.181602    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" event={"ID":"ad010e63-a217-403f-b132-908263363c9f","Type":"ContainerDied","Data":"4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338"}
Oct 29 23:09:20.181721 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.181652    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9" event={"ID":"ad010e63-a217-403f-b132-908263363c9f","Type":"ContainerDied","Data":"afcf6f68e4f5a9707996c2a19489715c9e9fa422873a5212821b0cf5897e0a23"}
Oct 29 23:09:20.181721 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.181680    4742 scope.go:122] "RemoveContainer" containerID="4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338"
Oct 29 23:09:20.190908 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.190892    4742 scope.go:122] "RemoveContainer" containerID="30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e"
Oct 29 23:09:20.198423 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.198400    4742 scope.go:122] "RemoveContainer" containerID="4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338"
Oct 29 23:09:20.198692 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:09:20.198673    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338\": container with ID starting with 4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338 not found: ID does not exist" containerID="4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338"
Oct 29 23:09:20.198757 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.198702    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338"} err="failed to get container status \"4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338\": rpc error: code = NotFound desc = could not find container \"4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338\": container with ID starting with 4b87d5fb14750797fe7021777900bd1441220f1cae13cf5ab20b2c2dd41ad338 not found: ID does not exist"
Oct 29 23:09:20.198757 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.198720    4742 scope.go:122] "RemoveContainer" containerID="30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e"
Oct 29 23:09:20.198970 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:09:20.198952    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e\": container with ID starting with 30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e not found: ID does not exist" containerID="30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e"
Oct 29 23:09:20.199016 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.198977    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e"} err="failed to get container status \"30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e\": rpc error: code = NotFound desc = could not find container \"30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e\": container with ID starting with 30a876cf7a3fa99027fecf63fc777ab69c7fe45f575fa627f8cef2919421910e not found: ID does not exist"
Oct 29 23:09:20.239789 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.239756    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"]
Oct 29 23:09:20.243653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.243636    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2b2738-openshift-default-5794cc55bfn94k9"]
Oct 29 23:09:20.989250 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:20.989211    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad010e63-a217-403f-b132-908263363c9f" path="/var/lib/kubelet/pods/ad010e63-a217-403f-b132-908263363c9f/volumes"
Oct 29 23:09:23.122080 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:09:23.122027    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:09:24.037650 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:24.037622    4742 scope.go:122] "RemoveContainer" containerID="39d570d8f251de25f1c06e586631a6e147fad37e61b2b926d7be8469d8f44ebe"
Oct 29 23:09:24.045296 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:24.045265    4742 scope.go:122] "RemoveContainer" containerID="7b4b382372f34cb2f7c404b389d5ac2baf8e141ad6fed4dd5bc5eea04e6910c5"
Oct 29 23:09:24.052481 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:24.052463    4742 scope.go:122] "RemoveContainer" containerID="958757d7eb7f7c4fa255058dfb0c4657b61fbe8673d7faf28332964523a06121"
Oct 29 23:09:24.766501 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:24.766468    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb" podUID="561f1fd1-81db-4d31-a3e9-568814146c5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:09:29.147230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.147205    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:09:29.241627 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.241586    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561f1fd1-81db-4d31-a3e9-568814146c5a-kube-api-access-dbwk2\" (UniqueName: \"kubernetes.io/projected/561f1fd1-81db-4d31-a3e9-568814146c5a-kube-api-access-dbwk2\") pod \"561f1fd1-81db-4d31-a3e9-568814146c5a\" (UID: \"561f1fd1-81db-4d31-a3e9-568814146c5a\") "
Oct 29 23:09:29.245336 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.244360    4742 generic.go:356] "Generic (PLEG): container finished" podID="561f1fd1-81db-4d31-a3e9-568814146c5a" containerID="c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687" exitCode=0
Oct 29 23:09:29.245336 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.245329    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561f1fd1-81db-4d31-a3e9-568814146c5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561f1fd1-81db-4d31-a3e9-568814146c5a-plugins-config-volume\") pod \"561f1fd1-81db-4d31-a3e9-568814146c5a\" (UID: \"561f1fd1-81db-4d31-a3e9-568814146c5a\") "
Oct 29 23:09:29.245774 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.244581    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"
Oct 29 23:09:29.246308 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.244404    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb" event={"ID":"561f1fd1-81db-4d31-a3e9-568814146c5a","Type":"ContainerDied","Data":"c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687"}
Oct 29 23:09:29.246529 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.246509    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb" event={"ID":"561f1fd1-81db-4d31-a3e9-568814146c5a","Type":"ContainerDied","Data":"5a7498c28f2d148838d80f0545d55afa357ed19204b118a539400384e88e779f"}
Oct 29 23:09:29.246649 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.246635    4742 scope.go:122] "RemoveContainer" containerID="c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687"
Oct 29 23:09:29.246844 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.246640    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561f1fd1-81db-4d31-a3e9-568814146c5a-plugins-config-volume" pod "561f1fd1-81db-4d31-a3e9-568814146c5a" (UID: "561f1fd1-81db-4d31-a3e9-568814146c5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:29.247495 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.247467    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561f1fd1-81db-4d31-a3e9-568814146c5a-kube-api-access-dbwk2" pod "561f1fd1-81db-4d31-a3e9-568814146c5a" (UID: "561f1fd1-81db-4d31-a3e9-568814146c5a"). InnerVolumeSpecName "kube-api-access-dbwk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:29.332397 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.332264    4742 scope.go:122] "RemoveContainer" containerID="c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687"
Oct 29 23:09:29.334545 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:09:29.334458    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687\": container with ID starting with c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687 not found: ID does not exist" containerID="c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687"
Oct 29 23:09:29.334545 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.334509    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687"} err="failed to get container status \"c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687\": rpc error: code = NotFound desc = could not find container \"c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687\": container with ID starting with c2f93d607bf046ced81dc70262f442ef9adab4d67e0972b43c6f27246cabb687 not found: ID does not exist"
Oct 29 23:09:29.346693 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.346668    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561f1fd1-81db-4d31-a3e9-568814146c5a-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.346693 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.346694    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbwk2\" (UniqueName: \"kubernetes.io/projected/561f1fd1-81db-4d31-a3e9-568814146c5a-kube-api-access-dbwk2\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.629691 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.629614    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"]
Oct 29 23:09:29.631141 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:29.631109    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2b2738-7f59c94685l2hb"]
Oct 29 23:09:30.989684 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:09:30.989646    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561f1fd1-81db-4d31-a3e9-568814146c5a" path="/var/lib/kubelet/pods/561f1fd1-81db-4d31-a3e9-568814146c5a/volumes"
Oct 29 23:09:53.122002 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:09:53.121960    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:10:23.121635 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:10:23.121598    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:10:53.122819 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:10:53.122781    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:11:23.123184 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:11:23.123146    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:11:32.625740 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.625708    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"]
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626068    4742 cpu_manager.go:414] "containerMap: removing container" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626080    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626116    4742 cpu_manager.go:414] "containerMap: removing container" podUID="e0aa1662-4655-4599-bb90-2166d581af0b" containerName="revoke-keys"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626122    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0aa1662-4655-4599-bb90-2166d581af0b" containerName="revoke-keys"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626135    4742 cpu_manager.go:414] "containerMap: removing container" podUID="561f1fd1-81db-4d31-a3e9-568814146c5a" containerName="payload-processing"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626140    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561f1fd1-81db-4d31-a3e9-568814146c5a" containerName="payload-processing"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626149    4742 cpu_manager.go:414] "containerMap: removing container" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626154    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626171    4742 cpu_manager.go:414] "containerMap: removing container" podUID="3d27c127-787d-4c02-8b5b-8f33a2dd9eee" containerName="main"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626177    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d27c127-787d-4c02-8b5b-8f33a2dd9eee" containerName="main"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626233    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0aa1662-4655-4599-bb90-2166d581af0b" containerName="revoke-keys"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626244    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="561f1fd1-81db-4d31-a3e9-568814146c5a" containerName="payload-processing"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626253    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626263    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d27c127-787d-4c02-8b5b-8f33a2dd9eee" containerName="main"
Oct 29 23:11:32.626274 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.626272    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad010e63-a217-403f-b132-908263363c9f" containerName="istio-proxy"
Oct 29 23:11:32.629904 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.629879    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.648277 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.648246    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a545e2-openshift-default-dockercfg-qq4wz\""
Oct 29 23:11:32.661699 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.661668    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"]
Oct 29 23:11:32.733781 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.733743    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-podinfo\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.733957 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.733801    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-token\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.733957 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.733884    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f6d4ae-e719-4d36-a930-3ed01770330f-istiod-ca-cert\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.733957 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.733922    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-certs\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.734061 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.733955    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-credential-socket\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.734061 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.734039    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-envoy\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.734155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.734102    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bcxj\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-kube-api-access-5bcxj\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.734155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.734125    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-socket\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.734155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.734145    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-data\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835107 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835036    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-podinfo\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835320 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835213    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-token\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835320 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835274    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f6d4ae-e719-4d36-a930-3ed01770330f-istiod-ca-cert\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835320 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835310    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-certs\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835491 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835341    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-credential-socket\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835491 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835380    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-envoy\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835491 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835430    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bcxj\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-kube-api-access-5bcxj\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835491 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835459    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-socket\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835491 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835488    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-data\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835761 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835710    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-credential-socket\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835829 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835777    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-certs\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835871    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-data\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.835990 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835963    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-socket\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.836135 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.835997    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f6d4ae-e719-4d36-a930-3ed01770330f-istiod-ca-cert\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.837661 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.837638    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-podinfo\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.837839 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.837818    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-envoy\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.852758 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.852720    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bcxj\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-kube-api-access-5bcxj\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.853178 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.853149    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-token\") pod \"e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") " pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:32.944503 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:32.944425    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:33.252833 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:33.252806    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"]
Oct 29 23:11:33.254734 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:11:33.254703    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f6d4ae_e719_4d36_a930_3ed01770330f.slice/crio-a8ddb6159950e153b36cf1c81350e1400fa5589ae8c555bdcf227bf0cb457793 WatchSource:0}: Error finding container a8ddb6159950e153b36cf1c81350e1400fa5589ae8c555bdcf227bf0cb457793: Status 404 returned error can't find the container with id a8ddb6159950e153b36cf1c81350e1400fa5589ae8c555bdcf227bf0cb457793
Oct 29 23:11:33.256801 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:33.256772    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:11:33.256881 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:33.256833    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:11:33.256881 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:33.256864    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:11:33.790650 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:33.790614    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" event={"ID":"10f6d4ae-e719-4d36-a930-3ed01770330f","Type":"ContainerStarted","Data":"dc03251ea670f625925f31d29272a07d904da7ecba139da9459488c3ea91ff07"}
Oct 29 23:11:33.791063 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:33.790672    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" event={"ID":"10f6d4ae-e719-4d36-a930-3ed01770330f","Type":"ContainerStarted","Data":"a8ddb6159950e153b36cf1c81350e1400fa5589ae8c555bdcf227bf0cb457793"}
Oct 29 23:11:33.849421 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:33.849370    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podStartSLOduration=1.849356226 podStartE2EDuration="1.849356226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:33.838249684 +0000 UTC m=+1750.978991268" watchObservedRunningTime="2026-10-29 23:11:33.849356226 +0000 UTC m=+1750.990097809"
Oct 29 23:11:33.944695 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:33.944659    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:34.950471 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:34.950438    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:35.800720 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:35.800643    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:35.801530 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:35.801509    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:38.149762 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.149726    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"]
Oct 29 23:11:38.153488 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.153467    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:38.174362 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.174327    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"]
Oct 29 23:11:38.197837 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.197812    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-848222af-dockercfg-sqs64\""
Oct 29 23:11:38.213992 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.213963    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a545e2\""
Oct 29 23:11:38.244717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.244683    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"]
Oct 29 23:11:38.248140 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.248121    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:38.267539 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.267510    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"]
Oct 29 23:11:38.286366 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.286332    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl\" (UID: \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:38.286546 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.286434    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfx2q\" (UniqueName: \"kubernetes.io/projected/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-kube-api-access-cfx2q\") pod \"payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl\" (UID: \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:38.388805 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.387723    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6k5\" (UniqueName: \"kubernetes.io/projected/80ed50b8-f50e-451a-b3de-6e26062ee873-kube-api-access-4h6k5\") pod \"payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl\" (UID: \"80ed50b8-f50e-451a-b3de-6e26062ee873\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:38.388805 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.387781    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfx2q\" (UniqueName: \"kubernetes.io/projected/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-kube-api-access-cfx2q\") pod \"payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl\" (UID: \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:38.388805 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.387950    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ed50b8-f50e-451a-b3de-6e26062ee873-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl\" (UID: \"80ed50b8-f50e-451a-b3de-6e26062ee873\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:38.388805 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.388144    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl\" (UID: \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:38.389259 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.389230    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl\" (UID: \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:38.405920 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.405852    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfx2q\" (UniqueName: \"kubernetes.io/projected/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-kube-api-access-cfx2q\") pod \"payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl\" (UID: \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:38.466887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.466850    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:38.489306 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.489276    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6k5\" (UniqueName: \"kubernetes.io/projected/80ed50b8-f50e-451a-b3de-6e26062ee873-kube-api-access-4h6k5\") pod \"payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl\" (UID: \"80ed50b8-f50e-451a-b3de-6e26062ee873\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:38.489451 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.489370    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ed50b8-f50e-451a-b3de-6e26062ee873-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl\" (UID: \"80ed50b8-f50e-451a-b3de-6e26062ee873\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:38.490245 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.490220    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ed50b8-f50e-451a-b3de-6e26062ee873-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl\" (UID: \"80ed50b8-f50e-451a-b3de-6e26062ee873\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:38.506982 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.506947    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6k5\" (UniqueName: \"kubernetes.io/projected/80ed50b8-f50e-451a-b3de-6e26062ee873-kube-api-access-4h6k5\") pod \"payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl\" (UID: \"80ed50b8-f50e-451a-b3de-6e26062ee873\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:38.560520 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.560489    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:38.765840 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.765790    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"]
Oct 29 23:11:38.768506 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:11:38.768481    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547b5bfa_4c76_45c7_bc5c_7bb7ef346df8.slice/crio-1dbfe2624e8ffc4b8447ced206dd021311eeb06df0b800630ef9b229a4d09c20 WatchSource:0}: Error finding container 1dbfe2624e8ffc4b8447ced206dd021311eeb06df0b800630ef9b229a4d09c20: Status 404 returned error can't find the container with id 1dbfe2624e8ffc4b8447ced206dd021311eeb06df0b800630ef9b229a4d09c20
Oct 29 23:11:38.824851 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.824812    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl" event={"ID":"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8","Type":"ContainerStarted","Data":"1dbfe2624e8ffc4b8447ced206dd021311eeb06df0b800630ef9b229a4d09c20"}
Oct 29 23:11:38.857641 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:38.857605    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"]
Oct 29 23:11:38.861018 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:11:38.860991    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ed50b8_f50e_451a_b3de_6e26062ee873.slice/crio-1c6113b0625e0d68f0f4e56865a180097e7208280137eb923f53bd1b5deb5ab6 WatchSource:0}: Error finding container 1c6113b0625e0d68f0f4e56865a180097e7208280137eb923f53bd1b5deb5ab6: Status 404 returned error can't find the container with id 1c6113b0625e0d68f0f4e56865a180097e7208280137eb923f53bd1b5deb5ab6
Oct 29 23:11:39.831367 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:39.831330    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl" event={"ID":"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8","Type":"ContainerStarted","Data":"de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b"}
Oct 29 23:11:39.831866 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:39.831397    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:39.832895 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:39.832871    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl" event={"ID":"80ed50b8-f50e-451a-b3de-6e26062ee873","Type":"ContainerStarted","Data":"be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b"}
Oct 29 23:11:39.832986 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:39.832920    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl" event={"ID":"80ed50b8-f50e-451a-b3de-6e26062ee873","Type":"ContainerStarted","Data":"1c6113b0625e0d68f0f4e56865a180097e7208280137eb923f53bd1b5deb5ab6"}
Oct 29 23:11:39.832986 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:39.832958    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:39.867040 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:39.866979    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl" podStartSLOduration=1.866965053 podStartE2EDuration="1.866965053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:39.86536231 +0000 UTC m=+1757.006103896" watchObservedRunningTime="2026-10-29 23:11:39.866965053 +0000 UTC m=+1757.007706637"
Oct 29 23:11:39.902226 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:39.902175    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl" podStartSLOduration=1.902161706 podStartE2EDuration="1.902161706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:39.899978468 +0000 UTC m=+1757.040720063" watchObservedRunningTime="2026-10-29 23:11:39.902161706 +0000 UTC m=+1757.042903290"
Oct 29 23:11:50.840926 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:50.840885    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:11:50.841564 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:50.840966    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:11:51.087386 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:51.087349    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": read tcp 10.134.0.2:36370->10.134.0.133:15021: read: connection reset by peer" start-of-body=
Oct 29 23:11:51.087588 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:51.087418    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": read tcp 10.134.0.2:36370->10.134.0.133:15021: read: connection reset by peer"
Oct 29 23:11:51.913256 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:51.913220    4742 generic.go:356] "Generic (PLEG): container finished" podID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerID="dc03251ea670f625925f31d29272a07d904da7ecba139da9459488c3ea91ff07" exitCode=0
Oct 29 23:11:51.913766 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:51.913300    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" event={"ID":"10f6d4ae-e719-4d36-a930-3ed01770330f","Type":"ContainerDied","Data":"dc03251ea670f625925f31d29272a07d904da7ecba139da9459488c3ea91ff07"}
Oct 29 23:11:51.913835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:51.913821    4742 scope.go:122] "RemoveContainer" containerID="dc03251ea670f625925f31d29272a07d904da7ecba139da9459488c3ea91ff07"
Oct 29 23:11:51.915005 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:51.914980    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:11:51.915072 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:51.915046    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:11:51.915131 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:51.915075    4742 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 23:11:52.919615 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:52.919582    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" event={"ID":"10f6d4ae-e719-4d36-a930-3ed01770330f","Type":"ContainerStarted","Data":"af5c0d79c0113dede87fdb332e08a63c9440c148095a00e2ffc8b5e93447a2d9"}
Oct 29 23:11:52.945244 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:52.945210    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:52.945426 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:52.945401    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:11:52.945498 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:52.945453    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:11:53.122873 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:11:53.122837    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:11:53.945525 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:53.945488    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:11:53.945951 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:53.945558    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:11:54.447661 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.447624    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"]
Oct 29 23:11:54.450261 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.450243    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.494284 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.494252    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-model-cache\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.494467 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.494299    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-dshm\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.494467 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.494389    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n5bf\" (UniqueName: \"kubernetes.io/projected/cc6984ee-39a2-42c0-b27f-f0da93873a4e-kube-api-access-8n5bf\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.494604 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.494492    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-home\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.494604 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.494553    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tls-certs\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.494722 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.494673    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tmp-dir\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.496220 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.496195    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a545e2\"/\"kube-root-ca.crt\""
Oct 29 23:11:54.496220 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.496208    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a545e2\"/\"openshift-service-ca.crt\""
Oct 29 23:11:54.496435 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.496236    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a545e2\"/\"simulator-single-main-a545e2-kserve-self-signed-certs\""
Oct 29 23:11:54.496435 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.496287    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a545e2\"/\"default-dockercfg-rjgbv\""
Oct 29 23:11:54.498655 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.498631    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"]
Oct 29 23:11:54.596137 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596071    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tmp-dir\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.596341 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596170    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-model-cache\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.596341 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596199    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-dshm\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.596341 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596224    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n5bf\" (UniqueName: \"kubernetes.io/projected/cc6984ee-39a2-42c0-b27f-f0da93873a4e-kube-api-access-8n5bf\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.596341 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596241    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-home\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.596633 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596369    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tls-certs\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.596633 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596496    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tmp-dir\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.596633 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596539    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-model-cache\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.596633 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.596588    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-home\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.598534 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.598516    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-dshm\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.598708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.598687    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tls-certs\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.613756 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.613733    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n5bf\" (UniqueName: \"kubernetes.io/projected/cc6984ee-39a2-42c0-b27f-f0da93873a4e-kube-api-access-8n5bf\") pod \"simulator-single-main-a545e2-kserve-787978c5d4-4vn9q\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") " pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.791651 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.791558    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:54.945625 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.945591    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:11:54.946023 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:54.945654    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:11:55.091755 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:55.091728    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"]
Oct 29 23:11:55.093460 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:11:55.093424    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc6984ee_39a2_42c0_b27f_f0da93873a4e.slice/crio-07e9e2dc4752439627cbad3f472a240f766bdb99bb2e49a4a848eb0f809d539e WatchSource:0}: Error finding container 07e9e2dc4752439627cbad3f472a240f766bdb99bb2e49a4a848eb0f809d539e: Status 404 returned error can't find the container with id 07e9e2dc4752439627cbad3f472a240f766bdb99bb2e49a4a848eb0f809d539e
Oct 29 23:11:55.934889 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:55.934855    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q" event={"ID":"cc6984ee-39a2-42c0-b27f-f0da93873a4e","Type":"ContainerStarted","Data":"d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc"}
Oct 29 23:11:55.935070 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:55.934900    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q" event={"ID":"cc6984ee-39a2-42c0-b27f-f0da93873a4e","Type":"ContainerStarted","Data":"07e9e2dc4752439627cbad3f472a240f766bdb99bb2e49a4a848eb0f809d539e"}
Oct 29 23:11:55.945612 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:55.945587    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:11:55.945729 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:55.945636    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:11:55.976108 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:55.971774    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q" podStartSLOduration=1.9717539259999999 podStartE2EDuration="1.971753926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:55.970848042 +0000 UTC m=+1773.111589629" watchObservedRunningTime="2026-10-29 23:11:55.971753926 +0000 UTC m=+1773.112495511"
Oct 29 23:11:56.795206 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:56.793800    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:11:56.944876 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:56.944835    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 29 23:11:56.945061 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:56.944899    4742 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 29 23:11:58.498846 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:58.498814    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:58.499945 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:58.499916    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:11:58.501079 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:11:58.501060    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:12:04.792344 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:04.792314    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:12:04.805016 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:04.804990    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:12:04.989692 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:04.989661    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:12:12.874945 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.874908    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"]
Oct 29 23:12:12.877691 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.877664    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:12.891303 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.891275    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:12:12.891435 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.891357    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:12:12.901619 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.901589    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:12:12.902127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.902063    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:12:12.904567 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.904542    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"]
Oct 29 23:12:12.982239 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.982209    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:12.982433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.982266    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:12.982433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:12.982401    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppx9\" (UniqueName: \"kubernetes.io/projected/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-kube-api-access-zppx9\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:13.082899 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:13.082866    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:13.083103 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:13.082921    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:13.083103 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:13.082993    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppx9\" (UniqueName: \"kubernetes.io/projected/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-kube-api-access-zppx9\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:13.083706 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:13.083681    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:13.085314 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:13.085285    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:13.100717 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:13.100687    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppx9\" (UniqueName: \"kubernetes.io/projected/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-kube-api-access-zppx9\") pod \"maas-api-revoke-sub-b0dcffaf-k5bgw\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:13.192248 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:13.192157    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:13.485041 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:13.485012    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"]
Oct 29 23:12:13.489913 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:12:13.487181    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ffffd_ae14_4cd8_b284_6cf0fe70b1ca.slice/crio-d19e007fe1d1e891273e35400621fa3c6228456e9d9facc38dc024a959c7af18 WatchSource:0}: Error finding container d19e007fe1d1e891273e35400621fa3c6228456e9d9facc38dc024a959c7af18: Status 404 returned error can't find the container with id d19e007fe1d1e891273e35400621fa3c6228456e9d9facc38dc024a959c7af18
Oct 29 23:12:14.017470 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:14.017378    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw" event={"ID":"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca","Type":"ContainerStarted","Data":"c3bf54c46f99e30a2adea1c5944acdf74504a871129459975693ce3cea08512c"}
Oct 29 23:12:14.017470 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:14.017437    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw" event={"ID":"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca","Type":"ContainerStarted","Data":"d19e007fe1d1e891273e35400621fa3c6228456e9d9facc38dc024a959c7af18"}
Oct 29 23:12:14.053994 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:14.053941    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw" podStartSLOduration=2.053926659 podStartE2EDuration="2.053926659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:14.052571883 +0000 UTC m=+1791.193313468" watchObservedRunningTime="2026-10-29 23:12:14.053926659 +0000 UTC m=+1791.194668242"
Oct 29 23:12:15.022966 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:15.022929    4742 generic.go:356] "Generic (PLEG): container finished" podID="b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca" containerID="c3bf54c46f99e30a2adea1c5944acdf74504a871129459975693ce3cea08512c" exitCode=0
Oct 29 23:12:15.023448 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:15.023019    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw" event={"ID":"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca","Type":"ContainerDied","Data":"c3bf54c46f99e30a2adea1c5944acdf74504a871129459975693ce3cea08512c"}
Oct 29 23:12:16.155989 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.155961    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:16.316487 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.316399    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-openshift-ca\") pod \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") "
Oct 29 23:12:16.316664 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.316565    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-kube-api-access-zppx9\" (UniqueName: \"kubernetes.io/projected/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-kube-api-access-zppx9\") pod \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") "
Oct 29 23:12:16.316664 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.316616    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-xks-ca\") pod \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\" (UID: \"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca\") "
Oct 29 23:12:16.316775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.316755    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-openshift-ca" pod "b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca" (UID: "b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:16.316924 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.316897    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:16.318618 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.318597    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-xks-ca" pod "b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca" (UID: "b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:16.318725 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.318707    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-kube-api-access-zppx9" pod "b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca" (UID: "b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca"). InnerVolumeSpecName "kube-api-access-zppx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:16.417620 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.417577    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zppx9\" (UniqueName: \"kubernetes.io/projected/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-kube-api-access-zppx9\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:16.417620 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:16.417614    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:17.032712 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:17.032682    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"
Oct 29 23:12:17.032893 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:17.032712    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw" event={"ID":"b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca","Type":"ContainerDied","Data":"d19e007fe1d1e891273e35400621fa3c6228456e9d9facc38dc024a959c7af18"}
Oct 29 23:12:17.032893 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:17.032758    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d19e007fe1d1e891273e35400621fa3c6228456e9d9facc38dc024a959c7af18"
Oct 29 23:12:23.121978 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:12:23.121933    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:12:23.903571 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.903537    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"]
Oct 29 23:12:23.903967 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.903951    4742 cpu_manager.go:414] "containerMap: removing container" podUID="b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca" containerName="revoke-keys"
Oct 29 23:12:23.904020 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.903969    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca" containerName="revoke-keys"
Oct 29 23:12:23.904062 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.904035    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca" containerName="revoke-keys"
Oct 29 23:12:23.906072 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.906057    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:23.930080 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.930052    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:12:23.930302 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.930159    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:12:23.930302 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.930240    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:12:23.932174 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.932141    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:12:23.932343 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:23.932321    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"]
Oct 29 23:12:24.091684 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.091640    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.091869 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.091710    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.091869 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.091732    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndnz\" (UniqueName: \"kubernetes.io/projected/c1082679-9f4a-414d-8b1b-047086e678c2-kube-api-access-tndnz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.192922 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.192827    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.192922 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.192875    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.193400 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.192944    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndnz\" (UniqueName: \"kubernetes.io/projected/c1082679-9f4a-414d-8b1b-047086e678c2-kube-api-access-tndnz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.193641 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.193622    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.195228 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.195205    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.210609 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.210585    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndnz\" (UniqueName: \"kubernetes.io/projected/c1082679-9f4a-414d-8b1b-047086e678c2-kube-api-access-tndnz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.217437 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.217415    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:24.517816 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:24.517683    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"]
Oct 29 23:12:25.075347 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:25.075251    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw" event={"ID":"c1082679-9f4a-414d-8b1b-047086e678c2","Type":"ContainerStarted","Data":"1b69b83c92336e42b4274de92cec519b3e74b98f7228d2e8b78f1681d332a4c6"}
Oct 29 23:12:25.075347 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:25.075309    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw" event={"ID":"c1082679-9f4a-414d-8b1b-047086e678c2","Type":"ContainerStarted","Data":"ccf7364b4f4ecc9277e887565481bdc7b9bd916c90b6efa5bdc0839b0cd7ff16"}
Oct 29 23:12:25.110811 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:25.110756    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw" podStartSLOduration=2.110741589 podStartE2EDuration="2.110741589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:25.109393606 +0000 UTC m=+1802.250135191" watchObservedRunningTime="2026-10-29 23:12:25.110741589 +0000 UTC m=+1802.251483172"
Oct 29 23:12:26.087477 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:26.087437    4742 generic.go:356] "Generic (PLEG): container finished" podID="c1082679-9f4a-414d-8b1b-047086e678c2" containerID="1b69b83c92336e42b4274de92cec519b3e74b98f7228d2e8b78f1681d332a4c6" exitCode=0
Oct 29 23:12:26.087862 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:26.087476    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw" event={"ID":"c1082679-9f4a-414d-8b1b-047086e678c2","Type":"ContainerDied","Data":"1b69b83c92336e42b4274de92cec519b3e74b98f7228d2e8b78f1681d332a4c6"}
Oct 29 23:12:27.223747 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.223719    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:27.224737 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.224719    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-openshift-ca\") pod \"c1082679-9f4a-414d-8b1b-047086e678c2\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") "
Oct 29 23:12:27.224805 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.224791    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-xks-ca\") pod \"c1082679-9f4a-414d-8b1b-047086e678c2\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") "
Oct 29 23:12:27.225049 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.225030    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-openshift-ca" pod "c1082679-9f4a-414d-8b1b-047086e678c2" (UID: "c1082679-9f4a-414d-8b1b-047086e678c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:27.225183 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.225164    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:27.226732 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.226711    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-xks-ca" pod "c1082679-9f4a-414d-8b1b-047086e678c2" (UID: "c1082679-9f4a-414d-8b1b-047086e678c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:27.326363 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.326328    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1082679-9f4a-414d-8b1b-047086e678c2-kube-api-access-tndnz\" (UniqueName: \"kubernetes.io/projected/c1082679-9f4a-414d-8b1b-047086e678c2-kube-api-access-tndnz\") pod \"c1082679-9f4a-414d-8b1b-047086e678c2\" (UID: \"c1082679-9f4a-414d-8b1b-047086e678c2\") "
Oct 29 23:12:27.326542 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.326530    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1082679-9f4a-414d-8b1b-047086e678c2-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:27.328335 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.328301    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1082679-9f4a-414d-8b1b-047086e678c2-kube-api-access-tndnz" pod "c1082679-9f4a-414d-8b1b-047086e678c2" (UID: "c1082679-9f4a-414d-8b1b-047086e678c2"). InnerVolumeSpecName "kube-api-access-tndnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:27.427075 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:27.427000    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndnz\" (UniqueName: \"kubernetes.io/projected/c1082679-9f4a-414d-8b1b-047086e678c2-kube-api-access-tndnz\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:28.097710 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:28.097682    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"
Oct 29 23:12:28.097886 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:28.097714    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw" event={"ID":"c1082679-9f4a-414d-8b1b-047086e678c2","Type":"ContainerDied","Data":"ccf7364b4f4ecc9277e887565481bdc7b9bd916c90b6efa5bdc0839b0cd7ff16"}
Oct 29 23:12:28.097886 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:28.097756    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf7364b4f4ecc9277e887565481bdc7b9bd916c90b6efa5bdc0839b0cd7ff16"
Oct 29 23:12:28.674423 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:28.674392    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"]
Oct 29 23:12:34.426535 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:34.426491    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"]
Oct 29 23:12:34.461194 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:34.461159    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"]
Oct 29 23:12:35.996218 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:35.996173    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"]
Oct 29 23:12:36.006210 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:36.006179    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a545e2-z8rpw"]
Oct 29 23:12:36.990461 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:36.990424    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1082679-9f4a-414d-8b1b-047086e678c2" path="/var/lib/kubelet/pods/c1082679-9f4a-414d-8b1b-047086e678c2/volumes"
Oct 29 23:12:39.427255 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:39.427203    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl" podUID="80ed50b8-f50e-451a-b3de-6e26062ee873" containerName="payload-processing" containerID="cri-o://be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b" gracePeriod=25
Oct 29 23:12:39.461841 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:39.461747    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl" podUID="547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" containerName="payload-pre-processing" containerID="cri-o://de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b" gracePeriod=25
Oct 29 23:12:40.840230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:40.840193    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl" podUID="547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:12:40.840230 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:40.840223    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl" podUID="80ed50b8-f50e-451a-b3de-6e26062ee873" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:12:42.990623 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:42.990591    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"]
Oct 29 23:12:42.991134 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:42.990782    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy" containerID="cri-o://af5c0d79c0113dede87fdb332e08a63c9440c148095a00e2ffc8b5e93447a2d9" gracePeriod=30
Oct 29 23:12:43.501380 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.501346    4742 patch_prober.go:29] interesting pod/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:12:43.501565 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.501411    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:12:43.694130 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.694054    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q" podUID="cc6984ee-39a2-42c0-b27f-f0da93873a4e" containerName="main" containerID="cri-o://d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc" gracePeriod=45
Oct 29 23:12:43.941245 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.941219    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:12:43.981104 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981056    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-dshm\") pod \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") "
Oct 29 23:12:43.981283 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981140    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tmp-dir\") pod \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") "
Oct 29 23:12:43.981283 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981222    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-model-cache\") pod \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") "
Oct 29 23:12:43.981283 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981274    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-home\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-home\") pod \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") "
Oct 29 23:12:43.981451 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981314    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tls-certs\") pod \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") "
Oct 29 23:12:43.981451 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981369    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6984ee-39a2-42c0-b27f-f0da93873a4e-kube-api-access-8n5bf\" (UniqueName: \"kubernetes.io/projected/cc6984ee-39a2-42c0-b27f-f0da93873a4e-kube-api-access-8n5bf\") pod \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\" (UID: \"cc6984ee-39a2-42c0-b27f-f0da93873a4e\") "
Oct 29 23:12:43.981451 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981394    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tmp-dir" pod "cc6984ee-39a2-42c0-b27f-f0da93873a4e" (UID: "cc6984ee-39a2-42c0-b27f-f0da93873a4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:43.981605 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981497    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-model-cache" pod "cc6984ee-39a2-42c0-b27f-f0da93873a4e" (UID: "cc6984ee-39a2-42c0-b27f-f0da93873a4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:43.981605 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981528    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-home" pod "cc6984ee-39a2-42c0-b27f-f0da93873a4e" (UID: "cc6984ee-39a2-42c0-b27f-f0da93873a4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:43.981740 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981723    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:43.981793 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981741    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:43.981793 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.981750    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:43.984176 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.984146    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-dshm" pod "cc6984ee-39a2-42c0-b27f-f0da93873a4e" (UID: "cc6984ee-39a2-42c0-b27f-f0da93873a4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:43.985451 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.985425    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6984ee-39a2-42c0-b27f-f0da93873a4e-kube-api-access-8n5bf" pod "cc6984ee-39a2-42c0-b27f-f0da93873a4e" (UID: "cc6984ee-39a2-42c0-b27f-f0da93873a4e"). InnerVolumeSpecName "kube-api-access-8n5bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:43.991482 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:43.991456    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tls-certs" pod "cc6984ee-39a2-42c0-b27f-f0da93873a4e" (UID: "cc6984ee-39a2-42c0-b27f-f0da93873a4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:44.083254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.083171    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6984ee-39a2-42c0-b27f-f0da93873a4e-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:44.083254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.083201    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n5bf\" (UniqueName: \"kubernetes.io/projected/cc6984ee-39a2-42c0-b27f-f0da93873a4e-kube-api-access-8n5bf\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:44.083254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.083214    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6984ee-39a2-42c0-b27f-f0da93873a4e-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:44.172123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.172067    4742 generic.go:356] "Generic (PLEG): container finished" podID="cc6984ee-39a2-42c0-b27f-f0da93873a4e" containerID="d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc" exitCode=0
Oct 29 23:12:44.172289 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.172125    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q" event={"ID":"cc6984ee-39a2-42c0-b27f-f0da93873a4e","Type":"ContainerDied","Data":"d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc"}
Oct 29 23:12:44.172289 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.172171    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q" event={"ID":"cc6984ee-39a2-42c0-b27f-f0da93873a4e","Type":"ContainerDied","Data":"07e9e2dc4752439627cbad3f472a240f766bdb99bb2e49a4a848eb0f809d539e"}
Oct 29 23:12:44.172289 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.172133    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"
Oct 29 23:12:44.172289 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.172189    4742 scope.go:122] "RemoveContainer" containerID="d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc"
Oct 29 23:12:44.181310 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.181286    4742 scope.go:122] "RemoveContainer" containerID="d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc"
Oct 29 23:12:44.181569 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:12:44.181547    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc\": container with ID starting with d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc not found: ID does not exist" containerID="d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc"
Oct 29 23:12:44.181642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.181583    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc"} err="failed to get container status \"d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc\": rpc error: code = NotFound desc = could not find container \"d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc\": container with ID starting with d462b2614ab06664ccdba26b48cd6ac429702f1c3a16fce87ba95780c38aa5dc not found: ID does not exist"
Oct 29 23:12:44.218887 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.218860    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"]
Oct 29 23:12:44.226158 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.226129    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a545e2/simulator-single-main-a545e2-kserve-787978c5d4-4vn9q"]
Oct 29 23:12:44.989763 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:44.989728    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6984ee-39a2-42c0-b27f-f0da93873a4e" path="/var/lib/kubelet/pods/cc6984ee-39a2-42c0-b27f-f0da93873a4e/volumes"
Oct 29 23:12:48.199756 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.199724    4742 generic.go:356] "Generic (PLEG): container finished" podID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerID="af5c0d79c0113dede87fdb332e08a63c9440c148095a00e2ffc8b5e93447a2d9" exitCode=0
Oct 29 23:12:48.200197 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.199771    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" event={"ID":"10f6d4ae-e719-4d36-a930-3ed01770330f","Type":"ContainerDied","Data":"af5c0d79c0113dede87fdb332e08a63c9440c148095a00e2ffc8b5e93447a2d9"}
Oct 29 23:12:48.200197 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.199848    4742 scope.go:122] "RemoveContainer" containerID="dc03251ea670f625925f31d29272a07d904da7ecba139da9459488c3ea91ff07"
Oct 29 23:12:48.235894 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.235866    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:12:48.326462 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326378    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f6d4ae-e719-4d36-a930-3ed01770330f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f6d4ae-e719-4d36-a930-3ed01770330f-istiod-ca-cert\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.326462 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326416    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-kube-api-access-5bcxj\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-kube-api-access-5bcxj\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.326701 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326476    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-certs\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.326701 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326501    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-data\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.326701 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326521    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-token\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-token\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.326701 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326638    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-credential-socket\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.326909 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326757    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-envoy\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.326909 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326805    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-podinfo\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.326909 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326835    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-socket\") pod \"10f6d4ae-e719-4d36-a930-3ed01770330f\" (UID: \"10f6d4ae-e719-4d36-a930-3ed01770330f\") "
Oct 29 23:12:48.327066 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326818    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-certs" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:48.327066 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326866    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-data" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:48.327066 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326920    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f6d4ae-e719-4d36-a930-3ed01770330f-istiod-ca-cert" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:48.327066 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.326978    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-credential-socket" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:48.327296 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.327128    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-socket" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:48.327296 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.327285    4742 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:48.327382 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.327298    4742 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-data\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:48.327382 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.327306    4742 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-credential-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:48.327382 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.327319    4742 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-workload-socket\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:48.327382 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.327334    4742 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f6d4ae-e719-4d36-a930-3ed01770330f-istiod-ca-cert\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:48.329235 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.329209    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-envoy" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:12:48.329348 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.329243    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-kube-api-access-5bcxj" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "kube-api-access-5bcxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:48.329348 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.329251    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-token" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:48.329348 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.329255    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-podinfo" pod "10f6d4ae-e719-4d36-a930-3ed01770330f" (UID: "10f6d4ae-e719-4d36-a930-3ed01770330f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:12:48.428708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.428665    4742 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-envoy\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:48.428708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.428700    4742 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-podinfo\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:48.428708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.428713    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bcxj\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-kube-api-access-5bcxj\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:48.428965 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:48.428727    4742 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f6d4ae-e719-4d36-a930-3ed01770330f-istio-token\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:49.205532 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:49.205501    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"
Oct 29 23:12:49.205532 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:49.205520    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr" event={"ID":"10f6d4ae-e719-4d36-a930-3ed01770330f","Type":"ContainerDied","Data":"a8ddb6159950e153b36cf1c81350e1400fa5589ae8c555bdcf227bf0cb457793"}
Oct 29 23:12:49.206031 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:49.205569    4742 scope.go:122] "RemoveContainer" containerID="af5c0d79c0113dede87fdb332e08a63c9440c148095a00e2ffc8b5e93447a2d9"
Oct 29 23:12:49.261761 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:49.261689    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"]
Oct 29 23:12:49.271896 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:49.271873    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a545e2-openshift-default-5b8576b685nt6mr"]
Oct 29 23:12:50.839891 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:50.839848    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl" podUID="547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:12:50.840441 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:50.840376    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl" podUID="80ed50b8-f50e-451a-b3de-6e26062ee873" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:12:50.992185 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:50.992140    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" path="/var/lib/kubelet/pods/10f6d4ae-e719-4d36-a930-3ed01770330f/volumes"
Oct 29 23:12:53.124108 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:12:53.124041    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:12:59.702227 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.702200    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:12:59.724897 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.724875    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:12:59.792344 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.792313    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-plugins-config-volume\") pod \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\" (UID: \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\") "
Oct 29 23:12:59.792537 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.792405    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ed50b8-f50e-451a-b3de-6e26062ee873-kube-api-access-4h6k5\" (UniqueName: \"kubernetes.io/projected/80ed50b8-f50e-451a-b3de-6e26062ee873-kube-api-access-4h6k5\") pod \"80ed50b8-f50e-451a-b3de-6e26062ee873\" (UID: \"80ed50b8-f50e-451a-b3de-6e26062ee873\") "
Oct 29 23:12:59.792537 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.792440    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ed50b8-f50e-451a-b3de-6e26062ee873-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ed50b8-f50e-451a-b3de-6e26062ee873-plugins-config-volume\") pod \"80ed50b8-f50e-451a-b3de-6e26062ee873\" (UID: \"80ed50b8-f50e-451a-b3de-6e26062ee873\") "
Oct 29 23:12:59.792537 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.792481    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-kube-api-access-cfx2q\" (UniqueName: \"kubernetes.io/projected/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-kube-api-access-cfx2q\") pod \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\" (UID: \"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8\") "
Oct 29 23:12:59.792782 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.792760    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-plugins-config-volume" pod "547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" (UID: "547b5bfa-4c76-45c7-bc5c-7bb7ef346df8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:59.792933 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.792901    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ed50b8-f50e-451a-b3de-6e26062ee873-plugins-config-volume" pod "80ed50b8-f50e-451a-b3de-6e26062ee873" (UID: "80ed50b8-f50e-451a-b3de-6e26062ee873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:59.794583 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.794562    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-kube-api-access-cfx2q" pod "547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" (UID: "547b5bfa-4c76-45c7-bc5c-7bb7ef346df8"). InnerVolumeSpecName "kube-api-access-cfx2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:59.794668 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.794607    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ed50b8-f50e-451a-b3de-6e26062ee873-kube-api-access-4h6k5" pod "80ed50b8-f50e-451a-b3de-6e26062ee873" (UID: "80ed50b8-f50e-451a-b3de-6e26062ee873"). InnerVolumeSpecName "kube-api-access-4h6k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:59.893793 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.893758    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:59.893793 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.893788    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h6k5\" (UniqueName: \"kubernetes.io/projected/80ed50b8-f50e-451a-b3de-6e26062ee873-kube-api-access-4h6k5\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:59.893793 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.893799    4742 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ed50b8-f50e-451a-b3de-6e26062ee873-plugins-config-volume\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:12:59.894035 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:12:59.893809    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfx2q\" (UniqueName: \"kubernetes.io/projected/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8-kube-api-access-cfx2q\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:13:00.277164 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.277128    4742 generic.go:356] "Generic (PLEG): container finished" podID="80ed50b8-f50e-451a-b3de-6e26062ee873" containerID="be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b" exitCode=0
Oct 29 23:13:00.277362 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.277183    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"
Oct 29 23:13:00.277362 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.277208    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl" event={"ID":"80ed50b8-f50e-451a-b3de-6e26062ee873","Type":"ContainerDied","Data":"be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b"}
Oct 29 23:13:00.277362 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.277268    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl" event={"ID":"80ed50b8-f50e-451a-b3de-6e26062ee873","Type":"ContainerDied","Data":"1c6113b0625e0d68f0f4e56865a180097e7208280137eb923f53bd1b5deb5ab6"}
Oct 29 23:13:00.277362 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.277293    4742 scope.go:122] "RemoveContainer" containerID="be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b"
Oct 29 23:13:00.278551 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.278527    4742 generic.go:356] "Generic (PLEG): container finished" podID="547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" containerID="de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b" exitCode=0
Oct 29 23:13:00.278671 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.278570    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"
Oct 29 23:13:00.278671 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.278590    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl" event={"ID":"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8","Type":"ContainerDied","Data":"de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b"}
Oct 29 23:13:00.278671 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.278630    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl" event={"ID":"547b5bfa-4c76-45c7-bc5c-7bb7ef346df8","Type":"ContainerDied","Data":"1dbfe2624e8ffc4b8447ced206dd021311eeb06df0b800630ef9b229a4d09c20"}
Oct 29 23:13:00.287543 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.287523    4742 scope.go:122] "RemoveContainer" containerID="be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b"
Oct 29 23:13:00.287852 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:13:00.287824    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b\": container with ID starting with be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b not found: ID does not exist" containerID="be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b"
Oct 29 23:13:00.287920 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.287870    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b"} err="failed to get container status \"be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b\": rpc error: code = NotFound desc = could not find container \"be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b\": container with ID starting with be8a2dec4f492b22b829b6d3b292f6189f7566c54b26d70da5b999767b4f6f4b not found: ID does not exist"
Oct 29 23:13:00.287920 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.287894    4742 scope.go:122] "RemoveContainer" containerID="de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b"
Oct 29 23:13:00.296076 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.296056    4742 scope.go:122] "RemoveContainer" containerID="de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b"
Oct 29 23:13:00.296373 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:13:00.296356    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b\": container with ID starting with de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b not found: ID does not exist" containerID="de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b"
Oct 29 23:13:00.296424 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.296380    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b"} err="failed to get container status \"de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b\": rpc error: code = NotFound desc = could not find container \"de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b\": container with ID starting with de171ec0928ad703d1e273b223ea9f6794fb77fbce2fb019dea50b42bf8da21b not found: ID does not exist"
Oct 29 23:13:00.324362 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.324333    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"]
Oct 29 23:13:00.331278 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.331256    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a545e2-7d5984zkfl"]
Oct 29 23:13:00.363790 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.363757    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"]
Oct 29 23:13:00.368190 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.368164    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a545e2-7d84d9c5dqqfgl"]
Oct 29 23:13:00.989221 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.989188    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" path="/var/lib/kubelet/pods/547b5bfa-4c76-45c7-bc5c-7bb7ef346df8/volumes"
Oct 29 23:13:00.989613 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:00.989544    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ed50b8-f50e-451a-b3de-6e26062ee873" path="/var/lib/kubelet/pods/80ed50b8-f50e-451a-b3de-6e26062ee873/volumes"
Oct 29 23:13:04.272951 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.272905    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"]
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273260    4742 cpu_manager.go:414] "containerMap: removing container" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273273    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273286    4742 cpu_manager.go:414] "containerMap: removing container" podUID="c1082679-9f4a-414d-8b1b-047086e678c2" containerName="revoke-keys"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273291    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1082679-9f4a-414d-8b1b-047086e678c2" containerName="revoke-keys"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273300    4742 cpu_manager.go:414] "containerMap: removing container" podUID="80ed50b8-f50e-451a-b3de-6e26062ee873" containerName="payload-processing"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273305    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ed50b8-f50e-451a-b3de-6e26062ee873" containerName="payload-processing"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273314    4742 cpu_manager.go:414] "containerMap: removing container" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273319    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273328    4742 cpu_manager.go:414] "containerMap: removing container" podUID="cc6984ee-39a2-42c0-b27f-f0da93873a4e" containerName="main"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273332    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6984ee-39a2-42c0-b27f-f0da93873a4e" containerName="main"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273349    4742 cpu_manager.go:414] "containerMap: removing container" podUID="547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" containerName="payload-pre-processing"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273354    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" containerName="payload-pre-processing"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273415    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="547b5bfa-4c76-45c7-bc5c-7bb7ef346df8" containerName="payload-pre-processing"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273424    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273430    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1082679-9f4a-414d-8b1b-047086e678c2" containerName="revoke-keys"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273438    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ed50b8-f50e-451a-b3de-6e26062ee873" containerName="payload-processing"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273444    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f6d4ae-e719-4d36-a930-3ed01770330f" containerName="istio-proxy"
Oct 29 23:13:04.273523 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.273451    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6984ee-39a2-42c0-b27f-f0da93873a4e" containerName="main"
Oct 29 23:13:04.276284 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.276262    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.301808 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.301780    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-95f34a42-kserve-self-signed-certs\""
Oct 29 23:13:04.304370 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.304347    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"]
Oct 29 23:13:04.332495 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.332460    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-home\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.332642 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.332524    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmptj\" (UniqueName: \"kubernetes.io/projected/5710b81f-a7fe-4798-af62-c841db01574a-kube-api-access-wmptj\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.332696 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.332651    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-model-cache\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.332734 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.332713    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-tmp-dir\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.332776 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.332761    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5710b81f-a7fe-4798-af62-c841db01574a-tls-certs\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.332820 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.332796    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-dshm\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.433778 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.433742    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmptj\" (UniqueName: \"kubernetes.io/projected/5710b81f-a7fe-4798-af62-c841db01574a-kube-api-access-wmptj\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.433979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.433789    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-model-cache\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.433979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.433829    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-tmp-dir\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.433979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.433856    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5710b81f-a7fe-4798-af62-c841db01574a-tls-certs\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.433979 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.433880    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-dshm\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.434197 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.434010    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-home\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.434324 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.434305    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-tmp-dir\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.434383 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.434318    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-model-cache\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.434383 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.434354    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-home\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.436254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.436222    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-dshm\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.436724 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.436703    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5710b81f-a7fe-4798-af62-c841db01574a-tls-certs\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.458130 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.456150    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmptj\" (UniqueName: \"kubernetes.io/projected/5710b81f-a7fe-4798-af62-c841db01574a-kube-api-access-wmptj\") pod \"e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") " pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.588364 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.588325    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:04.884060 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:04.884036    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"]
Oct 29 23:13:04.886822 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:13:04.886793    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5710b81f_a7fe_4798_af62_c841db01574a.slice/crio-7874033662a36f3a6d17bfe83d021b0c77d3f516ea6312eda1a52848d3d3fba6 WatchSource:0}: Error finding container 7874033662a36f3a6d17bfe83d021b0c77d3f516ea6312eda1a52848d3d3fba6: Status 404 returned error can't find the container with id 7874033662a36f3a6d17bfe83d021b0c77d3f516ea6312eda1a52848d3d3fba6
Oct 29 23:13:05.306003 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:05.305912    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk" event={"ID":"5710b81f-a7fe-4798-af62-c841db01574a","Type":"ContainerStarted","Data":"277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6"}
Oct 29 23:13:05.306003 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:05.305957    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk" event={"ID":"5710b81f-a7fe-4798-af62-c841db01574a","Type":"ContainerStarted","Data":"7874033662a36f3a6d17bfe83d021b0c77d3f516ea6312eda1a52848d3d3fba6"}
Oct 29 23:13:05.345975 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:05.345923    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk" podStartSLOduration=1.345911441 podStartE2EDuration="1.345911441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:05.344954885 +0000 UTC m=+1842.485696470" watchObservedRunningTime="2026-10-29 23:13:05.345911441 +0000 UTC m=+1842.486653025"
Oct 29 23:13:05.588875 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:05.588847    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:14.589167 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:14.589127    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:14.601486 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:14.601458    4742 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:15.368318 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:15.368285    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:23.126608 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:13:23.126564    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:13:39.282155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:39.282117    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"]
Oct 29 23:13:46.025433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:46.025398    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"]
Oct 29 23:13:46.035772 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:46.035743    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9860ca44-ldxwn"]
Oct 29 23:13:46.989057 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:46.989023    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939ddfed-2017-445c-bd1e-6d8e222a348a" path="/var/lib/kubelet/pods/939ddfed-2017-445c-bd1e-6d8e222a348a/volumes"
Oct 29 23:13:53.178109 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:13:53.176291    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:13:54.304522 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.304475    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk" podUID="5710b81f-a7fe-4798-af62-c841db01574a" containerName="main" containerID="cri-o://277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6" gracePeriod=45
Oct 29 23:13:54.358341 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.358301    4742 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk" podUID="5710b81f-a7fe-4798-af62-c841db01574a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.139:8000/ready\": dial tcp 10.134.0.139:8000: connect: connection refused"
Oct 29 23:13:54.555445 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.555388    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:54.605433 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.605405    4742 generic.go:356] "Generic (PLEG): container finished" podID="5710b81f-a7fe-4798-af62-c841db01574a" containerID="277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6" exitCode=0
Oct 29 23:13:54.605603 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.605459    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk" event={"ID":"5710b81f-a7fe-4798-af62-c841db01574a","Type":"ContainerDied","Data":"277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6"}
Oct 29 23:13:54.605603 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.605467    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"
Oct 29 23:13:54.605603 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.605489    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk" event={"ID":"5710b81f-a7fe-4798-af62-c841db01574a","Type":"ContainerDied","Data":"7874033662a36f3a6d17bfe83d021b0c77d3f516ea6312eda1a52848d3d3fba6"}
Oct 29 23:13:54.605603 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.605507    4742 scope.go:122] "RemoveContainer" containerID="277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6"
Oct 29 23:13:54.613644 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.613615    4742 scope.go:122] "RemoveContainer" containerID="277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6"
Oct 29 23:13:54.613969 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:13:54.613934    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6\": container with ID starting with 277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6 not found: ID does not exist" containerID="277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6"
Oct 29 23:13:54.614073 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.613981    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6"} err="failed to get container status \"277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6\": rpc error: code = NotFound desc = could not find container \"277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6\": container with ID starting with 277e5cdadac831e1d08a2f7f286b7feb3808b097d5d2a76c0aae06d76f36e4c6 not found: ID does not exist"
Oct 29 23:13:54.714454 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714419    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5710b81f-a7fe-4798-af62-c841db01574a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5710b81f-a7fe-4798-af62-c841db01574a-tls-certs\") pod \"5710b81f-a7fe-4798-af62-c841db01574a\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") "
Oct 29 23:13:54.714653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714496    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5710b81f-a7fe-4798-af62-c841db01574a-kube-api-access-wmptj\" (UniqueName: \"kubernetes.io/projected/5710b81f-a7fe-4798-af62-c841db01574a-kube-api-access-wmptj\") pod \"5710b81f-a7fe-4798-af62-c841db01574a\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") "
Oct 29 23:13:54.714653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714525    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-dshm\") pod \"5710b81f-a7fe-4798-af62-c841db01574a\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") "
Oct 29 23:13:54.714653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714573    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-tmp-dir\") pod \"5710b81f-a7fe-4798-af62-c841db01574a\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") "
Oct 29 23:13:54.714653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714617    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-model-cache\") pod \"5710b81f-a7fe-4798-af62-c841db01574a\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") "
Oct 29 23:13:54.714865 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714657    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-home\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-home\") pod \"5710b81f-a7fe-4798-af62-c841db01574a\" (UID: \"5710b81f-a7fe-4798-af62-c841db01574a\") "
Oct 29 23:13:54.714913 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714878    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-tmp-dir" pod "5710b81f-a7fe-4798-af62-c841db01574a" (UID: "5710b81f-a7fe-4798-af62-c841db01574a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:54.714958 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714907    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-model-cache" pod "5710b81f-a7fe-4798-af62-c841db01574a" (UID: "5710b81f-a7fe-4798-af62-c841db01574a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:54.715011 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.714973    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-home" pod "5710b81f-a7fe-4798-af62-c841db01574a" (UID: "5710b81f-a7fe-4798-af62-c841db01574a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:54.715123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.715072    4742 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-tmp-dir\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:13:54.715185 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.715125    4742 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-model-cache\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:13:54.715185 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.715139    4742 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-home\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:13:54.716775 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.716755    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5710b81f-a7fe-4798-af62-c841db01574a-kube-api-access-wmptj" pod "5710b81f-a7fe-4798-af62-c841db01574a" (UID: "5710b81f-a7fe-4798-af62-c841db01574a"). InnerVolumeSpecName "kube-api-access-wmptj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:54.716850 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.716815    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5710b81f-a7fe-4798-af62-c841db01574a-tls-certs" pod "5710b81f-a7fe-4798-af62-c841db01574a" (UID: "5710b81f-a7fe-4798-af62-c841db01574a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:13:54.716850 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.716820    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-dshm" pod "5710b81f-a7fe-4798-af62-c841db01574a" (UID: "5710b81f-a7fe-4798-af62-c841db01574a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:54.816652 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.816575    4742 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5710b81f-a7fe-4798-af62-c841db01574a-tls-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:13:54.816652 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.816607    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmptj\" (UniqueName: \"kubernetes.io/projected/5710b81f-a7fe-4798-af62-c841db01574a-kube-api-access-wmptj\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:13:54.816652 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.816623    4742 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5710b81f-a7fe-4798-af62-c841db01574a-dshm\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:13:54.949675 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.949643    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"]
Oct 29 23:13:54.960060 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.960034    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-95f34a42-kserve-6c5b6c7f57-sf8gk"]
Oct 29 23:13:54.988714 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:13:54.988678    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5710b81f-a7fe-4798-af62-c841db01574a" path="/var/lib/kubelet/pods/5710b81f-a7fe-4798-af62-c841db01574a/volumes"
Oct 29 23:14:23.122720 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:14:23.122681    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:14:23.572213 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.572134    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"]
Oct 29 23:14:23.572517 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.572502    4742 cpu_manager.go:414] "containerMap: removing container" podUID="5710b81f-a7fe-4798-af62-c841db01574a" containerName="main"
Oct 29 23:14:23.572572 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.572517    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5710b81f-a7fe-4798-af62-c841db01574a" containerName="main"
Oct 29 23:14:23.572607 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.572594    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="5710b81f-a7fe-4798-af62-c841db01574a" containerName="main"
Oct 29 23:14:23.575397 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.575382    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.588668 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.588642    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:14:23.599402 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.599371    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:14:23.599542 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.599469    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 29 23:14:23.599609 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.599568    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:14:23.599708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.599673    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7l5lq\""
Oct 29 23:14:23.601548 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.601527    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"]
Oct 29 23:14:23.682925 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.682890    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-maas-api-tls\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.683149 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.682941    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsfh6\" (UniqueName: \"kubernetes.io/projected/1c786f14-ed66-4120-a3eb-0b66fd566932-kube-api-access-vsfh6\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.683149 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.682962    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-metrics-certs\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.683149 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.683118    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c786f14-ed66-4120-a3eb-0b66fd566932-openshift-service-ca\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.783917 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.783887    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-maas-api-tls\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.783917 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.783936    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfh6\" (UniqueName: \"kubernetes.io/projected/1c786f14-ed66-4120-a3eb-0b66fd566932-kube-api-access-vsfh6\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.784192 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.783967    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-metrics-certs\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.784192 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.784009    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c786f14-ed66-4120-a3eb-0b66fd566932-openshift-service-ca\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.784744 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.784721    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c786f14-ed66-4120-a3eb-0b66fd566932-openshift-service-ca\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.786521 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.786500    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-metrics-certs\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.786521 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.786515    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-maas-api-tls\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.801024 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.801003    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfh6\" (UniqueName: \"kubernetes.io/projected/1c786f14-ed66-4120-a3eb-0b66fd566932-kube-api-access-vsfh6\") pod \"maas-api-5c48877c75-j5wdk\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:23.888136 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:23.888102    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:24.181399 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:24.181373    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"]
Oct 29 23:14:24.183871 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:14:24.183843    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c786f14_ed66_4120_a3eb_0b66fd566932.slice/crio-96997203419b63419fb18d4a4dbdcfad739eb89933fefc698bdc966019354f47 WatchSource:0}: Error finding container 96997203419b63419fb18d4a4dbdcfad739eb89933fefc698bdc966019354f47: Status 404 returned error can't find the container with id 96997203419b63419fb18d4a4dbdcfad739eb89933fefc698bdc966019354f47
Oct 29 23:14:24.185282 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:24.185266    4742 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:14:24.750223 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:24.750189    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk" event={"ID":"1c786f14-ed66-4120-a3eb-0b66fd566932","Type":"ContainerStarted","Data":"cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3"}
Oct 29 23:14:24.750419 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:24.750238    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk" event={"ID":"1c786f14-ed66-4120-a3eb-0b66fd566932","Type":"ContainerStarted","Data":"96997203419b63419fb18d4a4dbdcfad739eb89933fefc698bdc966019354f47"}
Oct 29 23:14:24.750419 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:24.750329    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:24.787499 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:24.787443    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk" podStartSLOduration=1.427115404 podStartE2EDuration="1.787425572s" totalImagesPullingTime="360.310168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:24.786196605 +0000 UTC m=+1921.926938212" watchObservedRunningTime="2026-10-29 23:14:24.787425572 +0000 UTC m=+1921.928167151"
Oct 29 23:14:30.760228 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:30.760200    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:31.701781 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.701750    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"]
Oct 29 23:14:31.701977 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.701932    4742 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk" podUID="1c786f14-ed66-4120-a3eb-0b66fd566932" containerName="maas-api" containerID="cri-o://cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3" gracePeriod=30
Oct 29 23:14:31.733067 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.733031    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"]
Oct 29 23:14:31.736578 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.736556    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.755640 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.755609    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"]
Oct 29 23:14:31.864872 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.864834    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be671fbf-587d-4021-b7a4-189c40743e8d-metrics-certs\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.865305 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.864890    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be671fbf-587d-4021-b7a4-189c40743e8d-maas-api-tls\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.865305 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.864985    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nvm\" (UniqueName: \"kubernetes.io/projected/be671fbf-587d-4021-b7a4-189c40743e8d-kube-api-access-w9nvm\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.865305 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.865029    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be671fbf-587d-4021-b7a4-189c40743e8d-openshift-service-ca\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.948644 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.948619    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:31.965714 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.965652    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be671fbf-587d-4021-b7a4-189c40743e8d-metrics-certs\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.965714 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.965690    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be671fbf-587d-4021-b7a4-189c40743e8d-maas-api-tls\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.965859 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.965812    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nvm\" (UniqueName: \"kubernetes.io/projected/be671fbf-587d-4021-b7a4-189c40743e8d-kube-api-access-w9nvm\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.965900 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.965881    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be671fbf-587d-4021-b7a4-189c40743e8d-openshift-service-ca\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.966516 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.966496    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be671fbf-587d-4021-b7a4-189c40743e8d-openshift-service-ca\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.968052 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.968031    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be671fbf-587d-4021-b7a4-189c40743e8d-metrics-certs\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.968254 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.968234    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be671fbf-587d-4021-b7a4-189c40743e8d-maas-api-tls\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:31.984471 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:31.984445    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nvm\" (UniqueName: \"kubernetes.io/projected/be671fbf-587d-4021-b7a4-189c40743e8d-kube-api-access-w9nvm\") pod \"maas-api-595cc648c-n5f4x\" (UID: \"be671fbf-587d-4021-b7a4-189c40743e8d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:32.066933 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.066902    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-metrics-certs\") pod \"1c786f14-ed66-4120-a3eb-0b66fd566932\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") "
Oct 29 23:14:32.066933 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.066940    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c786f14-ed66-4120-a3eb-0b66fd566932-kube-api-access-vsfh6\" (UniqueName: \"kubernetes.io/projected/1c786f14-ed66-4120-a3eb-0b66fd566932-kube-api-access-vsfh6\") pod \"1c786f14-ed66-4120-a3eb-0b66fd566932\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") "
Oct 29 23:14:32.067189 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.066993    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c786f14-ed66-4120-a3eb-0b66fd566932-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c786f14-ed66-4120-a3eb-0b66fd566932-openshift-service-ca\") pod \"1c786f14-ed66-4120-a3eb-0b66fd566932\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") "
Oct 29 23:14:32.067189 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.067125    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-maas-api-tls\") pod \"1c786f14-ed66-4120-a3eb-0b66fd566932\" (UID: \"1c786f14-ed66-4120-a3eb-0b66fd566932\") "
Oct 29 23:14:32.067484 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.067452    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c786f14-ed66-4120-a3eb-0b66fd566932-openshift-service-ca" pod "1c786f14-ed66-4120-a3eb-0b66fd566932" (UID: "1c786f14-ed66-4120-a3eb-0b66fd566932"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:32.069241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.069218    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-maas-api-tls" pod "1c786f14-ed66-4120-a3eb-0b66fd566932" (UID: "1c786f14-ed66-4120-a3eb-0b66fd566932"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:32.069241 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.069227    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c786f14-ed66-4120-a3eb-0b66fd566932-kube-api-access-vsfh6" pod "1c786f14-ed66-4120-a3eb-0b66fd566932" (UID: "1c786f14-ed66-4120-a3eb-0b66fd566932"). InnerVolumeSpecName "kube-api-access-vsfh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:32.069369 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.069234    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-metrics-certs" pod "1c786f14-ed66-4120-a3eb-0b66fd566932" (UID: "1c786f14-ed66-4120-a3eb-0b66fd566932"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:32.092464 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.092433    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:32.168845 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.168811    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-maas-api-tls\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:14:32.168845 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.168840    4742 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c786f14-ed66-4120-a3eb-0b66fd566932-metrics-certs\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:14:32.168845 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.168849    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsfh6\" (UniqueName: \"kubernetes.io/projected/1c786f14-ed66-4120-a3eb-0b66fd566932-kube-api-access-vsfh6\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:14:32.169170 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.168862    4742 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c786f14-ed66-4120-a3eb-0b66fd566932-openshift-service-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:14:32.384473 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.384444    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"]
Oct 29 23:14:32.386941 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:14:32.386907    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe671fbf_587d_4021_b7a4_189c40743e8d.slice/crio-9d88dc6d2682f416e2b8c481f856abc93ea8518184bac3c83d9786a9e447bcae WatchSource:0}: Error finding container 9d88dc6d2682f416e2b8c481f856abc93ea8518184bac3c83d9786a9e447bcae: Status 404 returned error can't find the container with id 9d88dc6d2682f416e2b8c481f856abc93ea8518184bac3c83d9786a9e447bcae
Oct 29 23:14:32.795307 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.795270    4742 generic.go:356] "Generic (PLEG): container finished" podID="1c786f14-ed66-4120-a3eb-0b66fd566932" containerID="cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3" exitCode=0
Oct 29 23:14:32.795503 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.795320    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"
Oct 29 23:14:32.795503 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.795354    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk" event={"ID":"1c786f14-ed66-4120-a3eb-0b66fd566932","Type":"ContainerDied","Data":"cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3"}
Oct 29 23:14:32.795503 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.795412    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk" event={"ID":"1c786f14-ed66-4120-a3eb-0b66fd566932","Type":"ContainerDied","Data":"96997203419b63419fb18d4a4dbdcfad739eb89933fefc698bdc966019354f47"}
Oct 29 23:14:32.795503 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.795437    4742 scope.go:122] "RemoveContainer" containerID="cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3"
Oct 29 23:14:32.797155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.797064    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x" event={"ID":"be671fbf-587d-4021-b7a4-189c40743e8d","Type":"ContainerStarted","Data":"a9bbb50a6c5cfc85e3220c3b3100dc1f147461ad86556be83662a0c340b932c1"}
Oct 29 23:14:32.797155 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.797128    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x" event={"ID":"be671fbf-587d-4021-b7a4-189c40743e8d","Type":"ContainerStarted","Data":"9d88dc6d2682f416e2b8c481f856abc93ea8518184bac3c83d9786a9e447bcae"}
Oct 29 23:14:32.797341 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.797218    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:32.805619 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.805599    4742 scope.go:122] "RemoveContainer" containerID="cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3"
Oct 29 23:14:32.805871 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:14:32.805849    4742 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3\": container with ID starting with cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3 not found: ID does not exist" containerID="cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3"
Oct 29 23:14:32.805945 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.805882    4742 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3"} err="failed to get container status \"cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3\": rpc error: code = NotFound desc = could not find container \"cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3\": container with ID starting with cbf3eeb6fd1246b148011c68a6403d6752ca879d6e3c60137884e25d680833a3 not found: ID does not exist"
Oct 29 23:14:32.845419 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.845345    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x" podStartSLOduration=1.570550329 podStartE2EDuration="1.84532853s" totalImagesPullingTime="274.778201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:32.845138099 +0000 UTC m=+1929.985879684" watchObservedRunningTime="2026-10-29 23:14:32.84532853 +0000 UTC m=+1929.986070115"
Oct 29 23:14:32.878563 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.878527    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"]
Oct 29 23:14:32.883884 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.883858    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j5wdk"]
Oct 29 23:14:32.990602 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:32.990514    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c786f14-ed66-4120-a3eb-0b66fd566932" path="/var/lib/kubelet/pods/1c786f14-ed66-4120-a3eb-0b66fd566932/volumes"
Oct 29 23:14:38.809968 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:14:38.809886    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-n5f4x"
Oct 29 23:14:53.123388 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:14:53.123348    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:15:06.021670 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.021636    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"]
Oct 29 23:15:06.022125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.021993    4742 cpu_manager.go:414] "containerMap: removing container" podUID="1c786f14-ed66-4120-a3eb-0b66fd566932" containerName="maas-api"
Oct 29 23:15:06.022125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.022004    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c786f14-ed66-4120-a3eb-0b66fd566932" containerName="maas-api"
Oct 29 23:15:06.022125 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.022077    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c786f14-ed66-4120-a3eb-0b66fd566932" containerName="maas-api"
Oct 29 23:15:06.025208 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.025190    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.038480 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.038456    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:15:06.048991 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.048958    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:15:06.051226 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.051197    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"]
Oct 29 23:15:06.171035 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.170988    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgwmt\" (UniqueName: \"kubernetes.io/projected/474cca33-91bf-4bb6-965a-56ee07ec4bc4-kube-api-access-bgwmt\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.171035 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.171038    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.171322 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.171061    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.272585 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.272495    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgwmt\" (UniqueName: \"kubernetes.io/projected/474cca33-91bf-4bb6-965a-56ee07ec4bc4-kube-api-access-bgwmt\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.272585 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.272537    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.272585 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.272566    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.273296 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.273273    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.274806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.274787    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.290698 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.290668    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgwmt\" (UniqueName: \"kubernetes.io/projected/474cca33-91bf-4bb6-965a-56ee07ec4bc4-kube-api-access-bgwmt\") pod \"maas-api-revoke-sub-0165d7d2-tz8s5\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.338335 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.338291    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:06.631820 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:06.631737    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"]
Oct 29 23:15:06.634038 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:15:06.634013    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474cca33_91bf_4bb6_965a_56ee07ec4bc4.slice/crio-7cd428b985c492d99972113d5662cb5432e6ddc89f599d64c618300354909c01 WatchSource:0}: Error finding container 7cd428b985c492d99972113d5662cb5432e6ddc89f599d64c618300354909c01: Status 404 returned error can't find the container with id 7cd428b985c492d99972113d5662cb5432e6ddc89f599d64c618300354909c01
Oct 29 23:15:07.058707 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:07.058665    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5" event={"ID":"474cca33-91bf-4bb6-965a-56ee07ec4bc4","Type":"ContainerStarted","Data":"718b495cf4349ebc5d8ecdb3f5665d1d304d7ef959e93f89437a0c8f7f859468"}
Oct 29 23:15:07.059151 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:07.058726    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5" event={"ID":"474cca33-91bf-4bb6-965a-56ee07ec4bc4","Type":"ContainerStarted","Data":"7cd428b985c492d99972113d5662cb5432e6ddc89f599d64c618300354909c01"}
Oct 29 23:15:07.106645 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:07.106586    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5" podStartSLOduration=2.106569702 podStartE2EDuration="2.106569702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:07.095626605 +0000 UTC m=+1964.236368190" watchObservedRunningTime="2026-10-29 23:15:07.106569702 +0000 UTC m=+1964.247311286"
Oct 29 23:15:08.064313 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:08.064279    4742 generic.go:356] "Generic (PLEG): container finished" podID="474cca33-91bf-4bb6-965a-56ee07ec4bc4" containerID="718b495cf4349ebc5d8ecdb3f5665d1d304d7ef959e93f89437a0c8f7f859468" exitCode=0
Oct 29 23:15:08.064740 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:08.064368    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5" event={"ID":"474cca33-91bf-4bb6-965a-56ee07ec4bc4","Type":"ContainerDied","Data":"718b495cf4349ebc5d8ecdb3f5665d1d304d7ef959e93f89437a0c8f7f859468"}
Oct 29 23:15:09.205022 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.204997    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:09.299461 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.299423    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-openshift-ca\") pod \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") "
Oct 29 23:15:09.299636 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.299485    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474cca33-91bf-4bb6-965a-56ee07ec4bc4-kube-api-access-bgwmt\" (UniqueName: \"kubernetes.io/projected/474cca33-91bf-4bb6-965a-56ee07ec4bc4-kube-api-access-bgwmt\") pod \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") "
Oct 29 23:15:09.299636 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.299540    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-xks-ca\") pod \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\" (UID: \"474cca33-91bf-4bb6-965a-56ee07ec4bc4\") "
Oct 29 23:15:09.299801 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.299779    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-openshift-ca" pod "474cca33-91bf-4bb6-965a-56ee07ec4bc4" (UID: "474cca33-91bf-4bb6-965a-56ee07ec4bc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:09.301585 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.301558    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474cca33-91bf-4bb6-965a-56ee07ec4bc4-kube-api-access-bgwmt" pod "474cca33-91bf-4bb6-965a-56ee07ec4bc4" (UID: "474cca33-91bf-4bb6-965a-56ee07ec4bc4"). InnerVolumeSpecName "kube-api-access-bgwmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:09.301585 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.301555    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-xks-ca" pod "474cca33-91bf-4bb6-965a-56ee07ec4bc4" (UID: "474cca33-91bf-4bb6-965a-56ee07ec4bc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:15:09.400942 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.400879    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.400942 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.400926    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgwmt\" (UniqueName: \"kubernetes.io/projected/474cca33-91bf-4bb6-965a-56ee07ec4bc4-kube-api-access-bgwmt\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:15:09.400942 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:09.400946    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474cca33-91bf-4bb6-965a-56ee07ec4bc4-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:15:10.074783 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:10.074750    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5"
Oct 29 23:15:10.074961 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:10.074751    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tz8s5" event={"ID":"474cca33-91bf-4bb6-965a-56ee07ec4bc4","Type":"ContainerDied","Data":"7cd428b985c492d99972113d5662cb5432e6ddc89f599d64c618300354909c01"}
Oct 29 23:15:10.074961 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:10.074899    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cd428b985c492d99972113d5662cb5432e6ddc89f599d64c618300354909c01"
Oct 29 23:15:17.049708 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.049671    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"]
Oct 29 23:15:17.050130 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.050025    4742 cpu_manager.go:414] "containerMap: removing container" podUID="474cca33-91bf-4bb6-965a-56ee07ec4bc4" containerName="revoke-keys"
Oct 29 23:15:17.050130 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.050035    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474cca33-91bf-4bb6-965a-56ee07ec4bc4" containerName="revoke-keys"
Oct 29 23:15:17.050130 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.050115    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="474cca33-91bf-4bb6-965a-56ee07ec4bc4" containerName="revoke-keys"
Oct 29 23:15:17.053425 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.053407    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.066865 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.066845    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:15:17.076888 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.076866    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:15:17.079577 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.079551    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"]
Oct 29 23:15:17.169681 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.169644    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882rd\" (UniqueName: \"kubernetes.io/projected/aab88e68-cb51-4bfe-be43-9781b73e0262-kube-api-access-882rd\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.169835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.169705    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.169835 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.169753    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.270362 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.270327    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882rd\" (UniqueName: \"kubernetes.io/projected/aab88e68-cb51-4bfe-be43-9781b73e0262-kube-api-access-882rd\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.270563 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.270376    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.270563 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.270407    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.271028 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.270993    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.272713 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.272691    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.288159 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.288138    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-882rd\" (UniqueName: \"kubernetes.io/projected/aab88e68-cb51-4bfe-be43-9781b73e0262-kube-api-access-882rd\") pod \"maas-api-revoke-sub-6b19338f-dchjh\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.366073 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.366044    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:17.660735 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:17.660705    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"]
Oct 29 23:15:17.664229 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:15:17.664204    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab88e68_cb51_4bfe_be43_9781b73e0262.slice/crio-8af25c3752686840829ec51e3ce13b0260eddc8d43c6ef589eb63dc4539da24f WatchSource:0}: Error finding container 8af25c3752686840829ec51e3ce13b0260eddc8d43c6ef589eb63dc4539da24f: Status 404 returned error can't find the container with id 8af25c3752686840829ec51e3ce13b0260eddc8d43c6ef589eb63dc4539da24f
Oct 29 23:15:18.111444 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:18.111406    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh" event={"ID":"aab88e68-cb51-4bfe-be43-9781b73e0262","Type":"ContainerStarted","Data":"8241aefda3755c9d03c28f5cbf4914bdf46a7df2df9576c4dfa8c70f97eda429"}
Oct 29 23:15:18.111837 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:18.111459    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh" event={"ID":"aab88e68-cb51-4bfe-be43-9781b73e0262","Type":"ContainerStarted","Data":"8af25c3752686840829ec51e3ce13b0260eddc8d43c6ef589eb63dc4539da24f"}
Oct 29 23:15:18.149144 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:18.149096    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh" podStartSLOduration=1.149068083 podStartE2EDuration="1.149068083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:18.148250322 +0000 UTC m=+1975.288991906" watchObservedRunningTime="2026-10-29 23:15:18.149068083 +0000 UTC m=+1975.289809667"
Oct 29 23:15:19.117651 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:19.117616    4742 generic.go:356] "Generic (PLEG): container finished" podID="aab88e68-cb51-4bfe-be43-9781b73e0262" containerID="8241aefda3755c9d03c28f5cbf4914bdf46a7df2df9576c4dfa8c70f97eda429" exitCode=0
Oct 29 23:15:19.118095 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:19.117707    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh" event={"ID":"aab88e68-cb51-4bfe-be43-9781b73e0262","Type":"ContainerDied","Data":"8241aefda3755c9d03c28f5cbf4914bdf46a7df2df9576c4dfa8c70f97eda429"}
Oct 29 23:15:20.253834 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.253807    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:20.397359 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.397279    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab88e68-cb51-4bfe-be43-9781b73e0262-kube-api-access-882rd\" (UniqueName: \"kubernetes.io/projected/aab88e68-cb51-4bfe-be43-9781b73e0262-kube-api-access-882rd\") pod \"aab88e68-cb51-4bfe-be43-9781b73e0262\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") "
Oct 29 23:15:20.397359 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.397350    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-xks-ca\") pod \"aab88e68-cb51-4bfe-be43-9781b73e0262\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") "
Oct 29 23:15:20.397555 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.397432    4742 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-openshift-ca\") pod \"aab88e68-cb51-4bfe-be43-9781b73e0262\" (UID: \"aab88e68-cb51-4bfe-be43-9781b73e0262\") "
Oct 29 23:15:20.397819 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.397790    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-openshift-ca" pod "aab88e68-cb51-4bfe-be43-9781b73e0262" (UID: "aab88e68-cb51-4bfe-be43-9781b73e0262"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:20.399530 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.399504    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-xks-ca" pod "aab88e68-cb51-4bfe-be43-9781b73e0262" (UID: "aab88e68-cb51-4bfe-be43-9781b73e0262"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:15:20.399530 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.399520    4742 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab88e68-cb51-4bfe-be43-9781b73e0262-kube-api-access-882rd" pod "aab88e68-cb51-4bfe-be43-9781b73e0262" (UID: "aab88e68-cb51-4bfe-be43-9781b73e0262"). InnerVolumeSpecName "kube-api-access-882rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:20.499131 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.499098    4742 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-882rd\" (UniqueName: \"kubernetes.io/projected/aab88e68-cb51-4bfe-be43-9781b73e0262-kube-api-access-882rd\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.499131 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.499127    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-xks-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.499131 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:20.499138    4742 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab88e68-cb51-4bfe-be43-9781b73e0262-maas-api-openshift-ca\") on node \"ip-10-0-139-130.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.127487 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:21.127451    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh" event={"ID":"aab88e68-cb51-4bfe-be43-9781b73e0262","Type":"ContainerDied","Data":"8af25c3752686840829ec51e3ce13b0260eddc8d43c6ef589eb63dc4539da24f"}
Oct 29 23:15:21.127487 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:21.127478    4742 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dchjh"
Oct 29 23:15:21.127699 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:21.127509    4742 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af25c3752686840829ec51e3ce13b0260eddc8d43c6ef589eb63dc4539da24f"
Oct 29 23:15:23.122912 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:15:23.122872    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:15:24.157238 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:24.157201    4742 scope.go:122] "RemoveContainer" containerID="877e240770147b1e30d89587d62241b964ae2b0f8e1dd06aa2afeeec997db7bd"
Oct 29 23:15:24.165263 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:24.165241    4742 scope.go:122] "RemoveContainer" containerID="a2a0080defba6a9bafd5c91282d22e4f63f037e17c141c00058fc7bcf44d9a93"
Oct 29 23:15:28.499434 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:15:28.499397    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d8467b9b9-5df8n_e9caaa6b-5c3b-4d63-a334-2f558eeff57a/authorino/0.log"
Oct 29 23:15:53.121638 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:15:53.121598    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:16:09.451653 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:09.451571    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-65h2w_d61643b2-b8ab-4251-95db-73e3969f05dd/manager/0.log"
Oct 29 23:16:09.653293 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:09.653261    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-ldmzq_ce0f0afa-5174-44d2-a80e-990ce4e0ec48/manager/0.log"
Oct 29 23:16:10.194986 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:10.194952    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-848f956548-dgrbz_f966c44c-b46f-4c0a-8461-6dccde541d26/manager/0.log"
Oct 29 23:16:10.604346 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:10.604318    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-nhwrp_c2b0ebb8-03da-4288-a806-a100d8e6ff1c/manager/0.log"
Oct 29 23:16:11.309942 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:11.309910    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675b698459-nn6hx_6d357a54-17bf-4d65-85db-134022425801/copy-manifests/0.log"
Oct 29 23:16:11.339888 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:11.339863    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675b698459-nn6hx_6d357a54-17bf-4d65-85db-134022425801/manager/0.log"
Oct 29 23:16:12.764100 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:12.764066    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-n5f4x_be671fbf-587d-4021-b7a4-189c40743e8d/maas-api/0.log"
Oct 29 23:16:12.951150 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:12.951121    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-tz8s5_474cca33-91bf-4bb6-965a-56ee07ec4bc4/revoke-keys/0.log"
Oct 29 23:16:13.506624 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:13.506593    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-dchjh_aab88e68-cb51-4bfe-be43-9781b73e0262/revoke-keys/0.log"
Oct 29 23:16:13.685275 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:13.685248    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b0dcffaf-k5bgw_b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca/revoke-keys/0.log"
Oct 29 23:16:15.953995 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:15.953965    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d8467b9b9-5df8n_e9caaa6b-5c3b-4d63-a334-2f558eeff57a/authorino/0.log"
Oct 29 23:16:18.088412 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:18.088382    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f44nnj_c571f4f4-be21-4c94-866b-cb67dd505862/istio-proxy/0.log"
Oct 29 23:16:18.530060 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:18.529989    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d6d655bb-2n6wv_d625bfef-ab90-429b-844f-d4551e8d1c8d/kube-auth-proxy/0.log"
Oct 29 23:16:19.254860 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:19.254828    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-c7pw8_87c1ed81-d887-47a9-a02e-04da08ca80b6/istio-proxy/0.log"
Oct 29 23:16:19.475219 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:19.475186    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-fbvvd_25ac9a2a-6abb-4ddb-aa72-b38f589e13b1/istio-proxy/0.log"
Oct 29 23:16:20.488509 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:20.488477    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-sk7lx_07991432-059a-40a2-9295-8d113f364302/istio-proxy/0.log"
Oct 29 23:16:20.895637 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:20.895605    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-tlj45_6ad493a0-72fe-4341-9ae5-1dd71d2d811f/payload-pre-processing/0.log"
Oct 29 23:16:22.273213 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:22.273176    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj_9d1ef9b7-cf83-4679-bd26-947a41a6cc71/main/0.log"
Oct 29 23:16:22.467946 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:22.467918    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rpq6f_145064cd-519a-4ccf-bdfa-b090c300b9b6/main/0.log"
Oct 29 23:16:22.840506 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:22.840478    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-t7bjt_f56a8bcc-2991-4a16-891e-fa69e277ca6a/main/0.log"
Oct 29 23:16:23.122900 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:16:23.122814    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:16:23.227146 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:23.227114    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw_b833cd5c-a426-4822-bcff-4965be0c17e4/main/0.log"
Oct 29 23:16:23.425977 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:23.425907    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7_2b671691-1df4-4fc2-9ba3-eb27464f3553/main/0.log"
Oct 29 23:16:24.320225 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:24.320196    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d8467b9b9-5df8n_e9caaa6b-5c3b-4d63-a334-2f558eeff57a/authorino/0.log"
Oct 29 23:16:33.739541 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:33.739507    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-65h2w_d61643b2-b8ab-4251-95db-73e3969f05dd/manager/0.log"
Oct 29 23:16:33.949067 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:33.949037    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-ldmzq_ce0f0afa-5174-44d2-a80e-990ce4e0ec48/manager/0.log"
Oct 29 23:16:34.408016 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:34.407983    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-848f956548-dgrbz_f966c44c-b46f-4c0a-8461-6dccde541d26/manager/0.log"
Oct 29 23:16:34.829127 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:34.829040    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-nhwrp_c2b0ebb8-03da-4288-a806-a100d8e6ff1c/manager/0.log"
Oct 29 23:16:35.582536 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:35.582503    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675b698459-nn6hx_6d357a54-17bf-4d65-85db-134022425801/copy-manifests/0.log"
Oct 29 23:16:35.613537 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:35.613509    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675b698459-nn6hx_6d357a54-17bf-4d65-85db-134022425801/manager/0.log"
Oct 29 23:16:36.809468 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:36.809435    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-n5f4x_be671fbf-587d-4021-b7a4-189c40743e8d/maas-api/0.log"
Oct 29 23:16:37.015004 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:37.014978    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-tz8s5_474cca33-91bf-4bb6-965a-56ee07ec4bc4/revoke-keys/0.log"
Oct 29 23:16:37.594208 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:37.594177    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-dchjh_aab88e68-cb51-4bfe-be43-9781b73e0262/revoke-keys/0.log"
Oct 29 23:16:37.777398 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:37.777373    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b0dcffaf-k5bgw_b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca/revoke-keys/0.log"
Oct 29 23:16:39.485138 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:39.485105    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d8467b9b9-5df8n_e9caaa6b-5c3b-4d63-a334-2f558eeff57a/authorino/0.log"
Oct 29 23:16:41.329503 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:41.329474    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f44nnj_c571f4f4-be21-4c94-866b-cb67dd505862/istio-proxy/0.log"
Oct 29 23:16:41.748995 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:41.748970    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d6d655bb-2n6wv_d625bfef-ab90-429b-844f-d4551e8d1c8d/kube-auth-proxy/0.log"
Oct 29 23:16:42.580712 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:42.580686    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-c7pw8_87c1ed81-d887-47a9-a02e-04da08ca80b6/istio-proxy/0.log"
Oct 29 23:16:42.810221 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:42.810196    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-fbvvd_25ac9a2a-6abb-4ddb-aa72-b38f589e13b1/istio-proxy/0.log"
Oct 29 23:16:43.901195 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:43.901168    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-sk7lx_07991432-059a-40a2-9295-8d113f364302/istio-proxy/0.log"
Oct 29 23:16:44.299186 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:44.299115    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-tlj45_6ad493a0-72fe-4341-9ae5-1dd71d2d811f/payload-pre-processing/0.log"
Oct 29 23:16:45.218963 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:45.218931    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7zlzj_9d1ef9b7-cf83-4679-bd26-947a41a6cc71/main/0.log"
Oct 29 23:16:45.401544 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:45.401515    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rpq6f_145064cd-519a-4ccf-bdfa-b090c300b9b6/main/0.log"
Oct 29 23:16:45.809374 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:45.809345    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-t7bjt_f56a8bcc-2991-4a16-891e-fa69e277ca6a/main/0.log"
Oct 29 23:16:46.202104 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:46.202053    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v92zw_b833cd5c-a426-4822-bcff-4965be0c17e4/main/0.log"
Oct 29 23:16:46.402483 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:16:46.402458    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-g98h7_2b671691-1df4-4fc2-9ba3-eb27464f3553/main/0.log"
Oct 29 23:16:53.121893 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:16:53.121849    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:17:17.029386 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:17:17.029343    4742 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"]
Oct 29 23:17:17.039634 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:17:17.039610    4742 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0dcffaf-k5bgw"]
Oct 29 23:17:18.989950 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:17:18.989915    4742 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca" path="/var/lib/kubelet/pods/b96ffffd-ae14-4cd8-b284-6cf0fe70b1ca/volumes"
Oct 29 23:17:23.122798 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:17:23.122758    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:17:24.793449 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:17:24.793417    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-848f956548-dgrbz_f966c44c-b46f-4c0a-8461-6dccde541d26/manager/0.log"
Oct 29 23:17:25.079972 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:17:25.079892    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-848f956548-dgrbz_f966c44c-b46f-4c0a-8461-6dccde541d26/manager/0.log"
Oct 29 23:17:53.122976 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:17:53.122893    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:18:12.935944 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:12.935913    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-848f956548-dgrbz_f966c44c-b46f-4c0a-8461-6dccde541d26/manager/0.log"
Oct 29 23:18:13.233579 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:13.233493    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-848f956548-dgrbz_f966c44c-b46f-4c0a-8461-6dccde541d26/manager/0.log"
Oct 29 23:18:14.526765 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.526730    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbmpv/must-gather-gnmjd"]
Oct 29 23:18:14.527174 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.527107    4742 cpu_manager.go:414] "containerMap: removing container" podUID="aab88e68-cb51-4bfe-be43-9781b73e0262" containerName="revoke-keys"
Oct 29 23:18:14.527174 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.527119    4742 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab88e68-cb51-4bfe-be43-9781b73e0262" containerName="revoke-keys"
Oct 29 23:18:14.527252 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.527201    4742 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab88e68-cb51-4bfe-be43-9781b73e0262" containerName="revoke-keys"
Oct 29 23:18:14.530358 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.530341    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmpv/must-gather-gnmjd"
Oct 29 23:18:14.558150 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.558121    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbmpv\"/\"openshift-service-ca.crt\""
Oct 29 23:18:14.558150 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.558124    4742 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rbmpv\"/\"default-dockercfg-wghv7\""
Oct 29 23:18:14.558336 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.558185    4742 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbmpv\"/\"kube-root-ca.crt\""
Oct 29 23:18:14.614710 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.614680    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpfp\" (UniqueName: \"kubernetes.io/projected/4254a00f-0688-442d-9f8d-595416716e04-kube-api-access-7xpfp\") pod \"must-gather-gnmjd\" (UID: \"4254a00f-0688-442d-9f8d-595416716e04\") " pod="openshift-must-gather-rbmpv/must-gather-gnmjd"
Oct 29 23:18:14.614878 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.614751    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4254a00f-0688-442d-9f8d-595416716e04-must-gather-output\") pod \"must-gather-gnmjd\" (UID: \"4254a00f-0688-442d-9f8d-595416716e04\") " pod="openshift-must-gather-rbmpv/must-gather-gnmjd"
Oct 29 23:18:14.629173 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.629141    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmpv/must-gather-gnmjd"]
Oct 29 23:18:14.715531 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.715489    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4254a00f-0688-442d-9f8d-595416716e04-must-gather-output\") pod \"must-gather-gnmjd\" (UID: \"4254a00f-0688-442d-9f8d-595416716e04\") " pod="openshift-must-gather-rbmpv/must-gather-gnmjd"
Oct 29 23:18:14.715713 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.715547    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpfp\" (UniqueName: \"kubernetes.io/projected/4254a00f-0688-442d-9f8d-595416716e04-kube-api-access-7xpfp\") pod \"must-gather-gnmjd\" (UID: \"4254a00f-0688-442d-9f8d-595416716e04\") " pod="openshift-must-gather-rbmpv/must-gather-gnmjd"
Oct 29 23:18:14.715832 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.715811    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4254a00f-0688-442d-9f8d-595416716e04-must-gather-output\") pod \"must-gather-gnmjd\" (UID: \"4254a00f-0688-442d-9f8d-595416716e04\") " pod="openshift-must-gather-rbmpv/must-gather-gnmjd"
Oct 29 23:18:14.733631 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.733600    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpfp\" (UniqueName: \"kubernetes.io/projected/4254a00f-0688-442d-9f8d-595416716e04-kube-api-access-7xpfp\") pod \"must-gather-gnmjd\" (UID: \"4254a00f-0688-442d-9f8d-595416716e04\") " pod="openshift-must-gather-rbmpv/must-gather-gnmjd"
Oct 29 23:18:14.844373 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:14.844336    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmpv/must-gather-gnmjd"
Oct 29 23:18:15.138844 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:15.138819    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmpv/must-gather-gnmjd"]
Oct 29 23:18:15.140880 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:18:15.140852    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4254a00f_0688_442d_9f8d_595416716e04.slice/crio-7643d732edc8d7e904ffb93038682e665387bb77a096e3ccc1c9b977cfbd5ce6 WatchSource:0}: Error finding container 7643d732edc8d7e904ffb93038682e665387bb77a096e3ccc1c9b977cfbd5ce6: Status 404 returned error can't find the container with id 7643d732edc8d7e904ffb93038682e665387bb77a096e3ccc1c9b977cfbd5ce6
Oct 29 23:18:15.916236 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:15.916202    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmpv/must-gather-gnmjd" event={"ID":"4254a00f-0688-442d-9f8d-595416716e04","Type":"ContainerStarted","Data":"7643d732edc8d7e904ffb93038682e665387bb77a096e3ccc1c9b977cfbd5ce6"}
Oct 29 23:18:16.929484 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:16.929452    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmpv/must-gather-gnmjd" event={"ID":"4254a00f-0688-442d-9f8d-595416716e04","Type":"ContainerStarted","Data":"5d67f01c055223b6c2a35770c58a1c79079a802206f698617fcb256c5868463c"}
Oct 29 23:18:16.929975 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:16.929495    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmpv/must-gather-gnmjd" event={"ID":"4254a00f-0688-442d-9f8d-595416716e04","Type":"ContainerStarted","Data":"1062d02e4d09e46b75d1d0c91165cbbacefcba8645de7e7097deeab3a812daab"}
Oct 29 23:18:16.965794 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:16.965742    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rbmpv/must-gather-gnmjd" podStartSLOduration=2.243660519 podStartE2EDuration="2.965724211s" totalImagesPullingTime="722.063692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:16.964449783 +0000 UTC m=+2154.105191367" watchObservedRunningTime="2026-10-29 23:18:16.965724211 +0000 UTC m=+2154.106465795"
Oct 29 23:18:19.122336 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:19.122304    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6vdmx_56ee256a-8f55-4c1d-bc82-f3cb5c3b2383/global-pull-secret-syncer/0.log"
Oct 29 23:18:19.989576 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:19.989538    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nfh24_549afed0-003b-43bf-8154-49e89fcba816/konnectivity-agent/0.log"
Oct 29 23:18:20.516063 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:20.516024    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-130.ec2.internal_67936acd58076934a9248bfd7519a6fc/haproxy/0.log"
Oct 29 23:18:23.088108 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:18:23.087756    4742 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4254a00f_0688_442d_9f8d_595416716e04.slice/crio-7643d732edc8d7e904ffb93038682e665387bb77a096e3ccc1c9b977cfbd5ce6\": RecentStats: unable to find data in memory cache]"
Oct 29 23:18:23.166122 ip-10-0-139-130 kubenswrapper[4742]: E1029 23:18:23.162502    4742 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5vrv4" containerName="extractor"
Oct 29 23:18:24.206110 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:24.204443    4742 scope.go:122] "RemoveContainer" containerID="c3bf54c46f99e30a2adea1c5944acdf74504a871129459975693ce3cea08512c"
Oct 29 23:18:26.478317 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:26.476682    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d8467b9b9-5df8n_e9caaa6b-5c3b-4d63-a334-2f558eeff57a/authorino/0.log"
Oct 29 23:18:28.592113 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.592050    4742 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"]
Oct 29 23:18:28.596463 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.596441    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.612271 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.612233    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"]
Oct 29 23:18:28.668649 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.668603    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-lib-modules\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.668984 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.668962    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-proc\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.669178 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.669161    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-sys\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.669380 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.669364    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-podres\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.678260 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.669530    4742 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hrs\" (UniqueName: \"kubernetes.io/projected/92e52a2a-1220-4d4c-873c-4e7c58b7be02-kube-api-access-v4hrs\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779646 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779611    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-lib-modules\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779646 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779646    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-proc\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779665    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-sys\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779702    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-podres\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779730    4742 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hrs\" (UniqueName: \"kubernetes.io/projected/92e52a2a-1220-4d4c-873c-4e7c58b7be02-kube-api-access-v4hrs\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779757    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-lib-modules\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779758    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-proc\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779808    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-sys\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.779954 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.779853    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92e52a2a-1220-4d4c-873c-4e7c58b7be02-podres\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.797426 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.797392    4742 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hrs\" (UniqueName: \"kubernetes.io/projected/92e52a2a-1220-4d4c-873c-4e7c58b7be02-kube-api-access-v4hrs\") pod \"perf-node-gather-daemonset-7r9rl\" (UID: \"92e52a2a-1220-4d4c-873c-4e7c58b7be02\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:28.921806 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:28.921725    4742 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:29.225831 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:29.225755    4742 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"]
Oct 29 23:18:29.228941 ip-10-0-139-130 kubenswrapper[4742]: W1029 23:18:29.228892    4742 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod92e52a2a_1220_4d4c_873c_4e7c58b7be02.slice/crio-35eea53d34b475f1abe0ce32759954401a463458436d01239975f9975e2f3c8a WatchSource:0}: Error finding container 35eea53d34b475f1abe0ce32759954401a463458436d01239975f9975e2f3c8a: Status 404 returned error can't find the container with id 35eea53d34b475f1abe0ce32759954401a463458436d01239975f9975e2f3c8a
Oct 29 23:18:30.053665 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:30.053632    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl" event={"ID":"92e52a2a-1220-4d4c-873c-4e7c58b7be02","Type":"ContainerStarted","Data":"ed3a201a38b5117fa0132e4bff29f6e86201a720cd1ecb511ce653b9f3fcc8e8"}
Oct 29 23:18:30.054333 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:30.053679    4742 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl" event={"ID":"92e52a2a-1220-4d4c-873c-4e7c58b7be02","Type":"ContainerStarted","Data":"35eea53d34b475f1abe0ce32759954401a463458436d01239975f9975e2f3c8a"}
Oct 29 23:18:30.054333 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:30.053766    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:30.090123 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:30.090053    4742 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl" podStartSLOduration=2.09003899 podStartE2EDuration="2.09003899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:30.089857463 +0000 UTC m=+2167.230599049" watchObservedRunningTime="2026-10-29 23:18:30.09003899 +0000 UTC m=+2167.230780616"
Oct 29 23:18:30.709614 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:30.709586    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgl57_7b17886d-b9f1-44b7-bcab-5fdebb8c3749/node-exporter/0.log"
Oct 29 23:18:30.827156 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:30.827127    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgl57_7b17886d-b9f1-44b7-bcab-5fdebb8c3749/kube-rbac-proxy/0.log"
Oct 29 23:18:30.947555 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:30.947524    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgl57_7b17886d-b9f1-44b7-bcab-5fdebb8c3749/init-textfile/0.log"
Oct 29 23:18:36.071353 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:36.071324    4742 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-7r9rl"
Oct 29 23:18:41.477246 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:41.477213    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f44nnj_c571f4f4-be21-4c94-866b-cb67dd505862/istio-proxy/0.log"
Oct 29 23:18:41.708333 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:41.708306    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzwmx_6ea35e55-0ac2-48aa-8e7d-217aae2f5445/dns/0.log"
Oct 29 23:18:41.826060 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:41.825988    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzwmx_6ea35e55-0ac2-48aa-8e7d-217aae2f5445/kube-rbac-proxy/0.log"
Oct 29 23:18:42.414529 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:42.414501    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xrt59_bffb5f7d-dcee-49b1-8efb-d0314fe27f5b/dns-node-resolver/0.log"
Oct 29 23:18:42.918239 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:42.918198    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d6d655bb-2n6wv_d625bfef-ab90-429b-844f-d4551e8d1c8d/kube-auth-proxy/0.log"
Oct 29 23:18:43.173624 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:43.173560    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-c7pw8_87c1ed81-d887-47a9-a02e-04da08ca80b6/istio-proxy/0.log"
Oct 29 23:18:43.236962 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:43.236937    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-fbvvd_25ac9a2a-6abb-4ddb-aa72-b38f589e13b1/istio-proxy/0.log"
Oct 29 23:18:43.562376 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:43.562299    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-sk7lx_07991432-059a-40a2-9295-8d113f364302/istio-proxy/0.log"
Oct 29 23:18:43.607212 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:43.607183    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d55dr_b117295a-3db5-4ec0-8858-43f561c7a075/node-ca/0.log"
Oct 29 23:18:43.684394 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:43.684369    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-tlj45_6ad493a0-72fe-4341-9ae5-1dd71d2d811f/payload-pre-processing/0.log"
Oct 29 23:18:45.720259 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:45.720192    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f44nnj_c571f4f4-be21-4c94-866b-cb67dd505862/istio-proxy/0.log"
Oct 29 23:18:46.260299 ip-10-0-139-130 kubenswrapper[4742]: I1029 23:18:46.260270    4742 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f44nnj_c571f4f4-be21-4c94-866b-cb67dd505862/istio-proxy/0.log"